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(57) Abstract: An MMS communication 
system for displaying images on a display 
terminal of a mobile or portable communica- 
tion device, the system comprising: an input 
adapted to receive pre-sowrce information^ 
transmitter adapted to transmit the pre-source 
information^ server adapted to receive the 
transmitted pre-source information and further 
adapted to convert the pre-source information 
to source information suitable for display on 
the display terminal; and a source transmitter 
adapted to transmit the source information to 
the display terminal. 
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MMS SYSTEM AND METHOD WITH PROTOCOL CONVERSION 
SUITABLE FOR MOBILE/PORTABLE HANDSET DISPLAY 

*F=DESCRIPTtON 

JfJL— R&iated Applications 
fl]Thls Application claims priority from co-pending U.S. Provisional 
Application Serial No. 60/299,745 filed June 22, 2QQ1, which is 
incorporated In its entirety by reference. 
JT.fi. f ffisltf 

r2]Thls disclosure teaches techniques related to an MMS (Multimedia 
Messaging System), including images, graphics, numerics, and text, 
suitable for display on the display of a mobile or portabie communication 
handset terminal. 

[3]To understand the disclosure better, the following definitions for technical 

terms used in this disclosure is provided: 
[431. EMS; Extended Messaging System, a source protocol used for EMS 

handsets, designed to encode multimedia messages, including images, 

graphics, numerics, animations, audio and formatted text. 
[5] 2. MMS; A pre-source (multimedia*formatting information) protocol used 

to encode types of messages, including images, graphics, numerics, and 

text, and transcoded for the display/phone speaker on various display 

terminals. Used in most Nokia handsets. 
f6]3. PM; Picture Messaging protocol, a graphic format (source protocol) 

used to display B/W images in Nokia handsets supporting tfie NSM per- 

source format. 

{714. Pre-source information; In this application, information, which may be 
a full multimedia message or some part thereof, which appears in a non~ 
source format and Is not coded in a source protocol. Pre-souree 
information refers to "packaged* multimedia content in a "raw* format 
such 

t&3a. A set of TCP/IP packets composing a MIME multipart message (could be 
an email message or an MMS MM1 message), where some parts of the 
message are media objects which need to be converted/transcoded, and 
some other parts (e.g. SMIL attachment) are presentation layer 
information relating to how the information has to be arranged and 
displayed. Figa, 24 and 25 illustrate this concept. 

t9]b, A block of SMS messages that together compose an EMS or a Nokia 
Smart Messaging (NSM) message and contain multiple media objects - 
pictures, ringtones, etc. These SMS messages are further encapsulated 
into the SMSC protocol which can be SMPP,UCP,CIMD etc. 

tlO] 5. Smart Messaging; A source protocol being developed by Nokia for 
Nodia handsets. This refers to everything defined in the NSM pre-source 
protocol and adds functionality for calendar events (vCalendar), electronic 
business cards (vCard) etc. 
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[11] 6, Source Information: In this application. Information, which may be a 
full multimedia message or some part thereof, which appears in a source 
format and which is coded in a source protocol. Typical source protocols 
are WBMP, EMS, and PM, but new protocols are being developed on an 
ongoing basis. Source protocols enable the display of messages on 
terminals with limit memory, processing, and display capabilities, such as 
those of mobile and portable radio communication handsets (i,e,, cellular 
telephones, land mobile radios, Instant Messaging terminals, radio 
enabled PDAs, and the like). On the other hand, source information 
constitutes media objects in some media format, e,g, a JPEG picture, an 
MP3 audio file, an AVI video etc. 

til] 7. Transcoding: To perform protocol conversion, either from one 
source protocol to another, or from a pre-souree protocol into a source 
protocol, 

t!3] 8s WAP: '"Wireless Application Protocol", one protocol used for what 
have been catted 2,5G cellular systems. In the cellular world, 1Q was the 
original set of analog cellular systems, iG has been mainly displaced by 
XG systems, which are low-speed digital systems, which a typical raw data 
rate of 9,5kbps, Operators are currently deploying what are known as 
2,5G systems, which are higher speed digital systems, expected to 
operate up 384kbps, 2,5G systems are expected to be replaced by 3G 
systems, which are higher speed digital systems promising speeds up to 
2Mbps, WAP is one of the chief manifestations of 2,5G systems, WAP is a 
pre-souree protocol, 

[14] 9, WBMP: The display protocol for handsets in the WAP system, 

^—Introduction 

CIS] The process of transcoding is not a new idea. Indeed, in forms the 
basis of communication systems. Even the conversion from analog to 
digital, or vice verse, is a form of transcoding. With the proliferation of 
higher speed digital cellular systems, the challenge and the problem of 
transcoding have become much greater. There is, as yet, no standard 
display protocol for higher speed communication terminals. Therefore, 
transcoding from one display protocol to another is required to insure that 
the receiving terminal will be able to display the transmuted. There is, 
however, no method or system to do this in such a way that the integrity 
and quality of the transmitted message will be maintained in the display 
terminal. Further, there is no method or system for transcoding non- 
source information into a source protocol suitable for display on a 
communication terminal, while maintaining the integrity and quality of the 
information which originally appeared in the non-source format There 
are transcoding systems and methods, to be sure, but they are primitive, 
and lose much of the quality of the source or pre-source information, even 
to the point where in some cases the displayed information in the display 
terminal Is not recognisable. What is required Is algorithms, a method, 
and a system, that will allow identification of the specific display 
characteristics of the target display terminal, and will also allow the source 
or pre-source information to be displayed on the target display terminal 
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with the maximum amount of integrity and quality in comparison to the 
pre-coded Information, 
ife=SUMMARY 

[16] The disclosed teachings provide fori 

[17] 1* Conversion of source information coded In a source format Into a 
protocol suitable for transmission to and display on the terminal A 
variety of new processing techniques are disclosed, Source information is 
typically coded in protocols such as WBMP (the protocol for Wireless 
/^plication Protocol, or *WAP*, systems), EMS* and PM* This information 
must be transeoded for display on different terminals, also using source 
protocols, but where the protocols and variations of the protocols are 
typically different between the input source and the display terminal 

[18] 2. Conversion of pre-source information, that is, Information which is 
coded but not in a source protocol, into a source protocol Par example, 
an ordinary digits* picture will be transcoded into the source protocol 
WBMP* tt will be appreciated that information in source protocol will then 
be transcoded again Into the target source protocol, as e^tplained in 
introductory point 1 immediately above, 

[19] An MM3 communication system for displaying images on a display 
terminal of a mobile or portable communication device, the system 
comprising : an input adapted to receive pre-source informations 
transmitter adapted to transmit the pre-source information^ server 
adapted to receive the transmitted pre-souree information and further 
adapted to convert the pre-source information to source information 
suitable for display on the display terminal; and a source transmitter 
adapted to transmit the source information to the display terminal. 



ttfe—BRIEF DESCRIPTION OF THE DRAWINGS 

[20] The above advantages of the disclosed teachings will become more 
apparent by describing In detail preferred embodiments thereof with 
reference to the attached drawings in which* 

[21] FIGS, 1*37 show various features of the dlsdosed teachings as 
described In the rest of this document 

l¥f=DETAILED DESCRIPTION 
W*As- Ovemit Artfcftecttm** 

122} An Implementation of the disclosed teachings ts shown in PIG,5* The 
structure includes the input devices 5*11-5*13 on the left, the server 5,2 
represented by the block In the middle, and the display devices 5*31-5*34 
shown on the right Information may be in source format, as is the case 
for the cellular telephone 5,1S (picture of two people) and the digital 
camera attached to a cellular telephone 5*13 (picture of the automobile). 
Or Information may be in a pre-source format, such as the cartoon of the 
man 5 V 1L At the server, the source information or pre-source 
Information is processed by a variety of components, adapted to 
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implement a variety of algorithms or techniques, which form an Integral 
part of the disclosed teachings, 
[23] Some of the components mentioned above perform at least the 
following tasks; 

[243 Format transcoding (from the pre-source information into source 

information, or from source information into other source Information 

suitable for display on the display terminal); 
[25] 2, Image adaptation, to adapt the Image to the particular display 

screen on the display terminal. (This is discussed in further detail in 

section IVG); 

[26] 3, Optimal compression for handset The display terminal cannot 
display all of the bits of the original information. The information must be 
compressed, both for transmission and for display* 

[27] 4, Photo enhancement; Specific sections of a photograph may be 
cutout and enhanced* (This Is discussed in further detail in section IVG); 

[28] 5, Content based processing, by which different aspects of a 
multimedia message are identified and processed differently. (This is 
discussed in further detail in section IVG); 

[29] 6. Recognition from images; This is allied to the photo enhancement 
algorithm. Different portions of an Image are recognized and "cutout* for 
enhancement (This is discussed in further detail in section IVG); 

[30] 7. Interfaces to 3rd Party Applications; Third party applications may be 
processed separately and sent to the display terminals, or may be added 
to the original information* In addition, if there are software packages 
with additional algorithms for additional processing of the source 
information, these may abe accessed and applied to the original 
information for eventual display on the display terminals* An example of 
an interface to a third party application is an XML-based interface over 
TCP/IP* Another example of such an interface would be and API in C++ 
or Java. By third party application we are referring to both external SW 
modules, and to VAS = value added services - e.g* a news service, a 
gaming platform far cellular phones (for example 
www.wirelessgames.com, www.cash-u.com), a photo album service. 
These applications wish to send MMS content and also in certain cases to 
perform special processing on such content prior to sending it. For a 
review of the special functionality please refer to section IV* H* 

[31] Another implementation of the disclosed teachings is shown in FIG.6* 
Examples of input sources, called "Content Sources*, 6,11-6*14, appear in 
the column at the right* However, the input sources are much broader 
than these pictures. At the bottom of the figure are various information 
devices 6,21-6,24 which can serve as both sources of information to the 
server 6.3, and also receivers of information processed by the server. 
These information sources can be WAP or i-Mode Phones, called *MMS 
Box" in the slide. (t-Mode phones are those that operate on i-Mode 2.SG 
cellular systems currently functioning in Japan. i-Mode phones will also 
operate on 3G systems expected to be introduced in Japan in late 2001 
and in 2002.) Picture messaging phones, operating with the PM protocol, 
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are portrayed In the "Picture Box*. Similarly, EMS phones, operating with 
the EMS protocol* are portrayed in the *EMS Box*. Finally, Email enabled 
phones are portrayed in the "E-mail Box*. In this implementation, 
information may be coded out of or coded into* any of the protocols show* 
including WAP phones (WBMP protocol), I-Mode Phones (the Japanese 
version of WAP), Picture Messeging Phones (PM protocol), EMS Compliant 
Phones (EMS protocol), and E-mail Capable Phones (POP3, SMTP, and 
IMAP4 protocols) 

[32] The server will receive, transeode, and optimize for display on a 
specific communication terminal, at least any of the following protocols; 
IP, SMPP, TCP/IP, POP3/SMTP/IMAP4/XML This is Illustrated in FIG,? 

[33] Using the transcoding several tasks are accomplished, for example; 

[34] lv The conversion from and into any of the various formats WBMP, PM, 
and EMS. 

[3S] 2. The conversion of formats on the fly when the user logs on to his or 

her MMS box (so that even if the user were to switch terminal devices, he 

or she would get the correct content, properly formatted). 
[36] 3. The conversion taking into account the exact parameters (such as, 

for example, screen size, pixel dimensions, etc) of the particular terminal 

and terminal display. 
[37] 4. Transcoding of source information into other source information 

suitable for display on a target terminal, maintaining integrity and quality 

of the original message, examples would be; 
[38] a. A 3PEG image is converted to a GIF image so that a phone with a 

WAP browser than can display GIF images will be able to view it. 
[39] b, A Nokia ringtone is converted to an EMS iMelody ringtone so that a 

non*Nokia phone can play it. 
[40] c. A video in MPEG1 format is converted to MPEG-4 Simple Visual 

profile so that an MMS compliant phone can display it, or to an animated 

GIF sequence so that a non-MMS compliant (legacy) phone can view it. 
[41] d„ Formatted text in the EMS format can be converted to an image or 

to HTML+text to preserve the formatting (underline, bold, letter size etc.). 
[42] It should be noted that both the EMS and NSM formats include not 

only images but also ringtones, animations and formatted text. This fact Is 

well documented in the H^IS/NMS standards documents for the last few 

years. 

[43] 5. Transcoding of non-source information into other source 

information, for later transcoding into source information for display on a 
specific target terminal, where all transcoding maintains the integrity and 
quality of the original message. 

[44] 6. Recognizing the specific display characteristics of a specific target 
display terminal, to enable the display of high quality messages, whatever 
the characteristics of the terminal. 

Wilh — An Example Implementation: Overview 
[45] RG.8 shorn another example implementation of the overall system. This implementation is called 
MMR. The MMR system allows users to send and receive messages containing text and images at least 
in the following formats / protocols: WAP; PM; EMS; MMS; E-MAIL; WEB; SMS. 
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[46] While the term message and information has been used in discussing the implementations in detail, it 
should be clear to a skilled artisan as to which message/information constitutes pre-souroe information 
and which constitutes source information according to the definitions for the same provided in the 
background. 

4^— User WAP Pages 

[47] The MMR provides a WAP based messaging application, allowing users to login to their personal 
messaging page- From this page users can view and send messages in variety of formats. The MMR 
sends the WAP recipient an SMS notification with a link to the newly received message, Mtematively , 
the MMR can send a WAP push message with the same link MMS recipients receive a notification to 
the new MMS, PM and EMS recipients receive the message directly. Email recipients receive an e-mail 
with an image attachment to their regular e-mail address. 

[48] It should be noted that WAP pages can contain multimedia for immediate integrated display (e,g. a 
WBMP image), but can also contain downloadable multimedia such as higher resolution images, 
ringtones, animations etc, , as part of e.g. the M-Services standard for downloadable media, 
S^User Web Pages 

[49] The MMR provides a WEB based portal for three major function. 

[50] 1. Users can register themselves to the system, by subrnittiog personal information as well as 

mformation about the model of their mobile device, 
[51] 2. A photo album application is provided for personal storage and sharing of images and audio tiles. 

Users can login to their personal accounts, view and send messages to mobile device, in the same 

manner as described above, 
[52] 3, Users can also login to an HTML based messaging page which allows them to view all meir received 

messages. Regardless of the format in which these messages were originally sent, these messages will 

be transcoded to be viewed on a normal web browser. This may allow users to view messages in a 

much higher quality than that seen on their mobile devices, 
fe^MMS Module 

[53]The MMR can send and receive MMS messages to and from mobile devices, mcoming MMS messages 
are parsed and transcoded for optimal display on the recipients device. Recipients of outgoing MMS 
messages receive a notification, allowing them to download the message from the MMS proxy. Other 
recipients receive the MMS message after it has been transcoded to WAP, PM , EMS „ SMS or E-Mail 
fe^Bmail Module 

[54] The MMR allows users to send E-mail messages with image attachments to mobile devices. It also 
allows mobile users to send B-Mails with image attachments to regular e-mail addresses. Incoming E- 
mails are parsed The e-mail subject js sent as the message text, while each of the image attachments in 
the original e-mail are transcoded for the mobile device. Depending on the amount of attachments, the 
recipient may receive several messages, and a format most suitable to his mobile device. Outgoing e- 
mail messages use SMTP to send the message text along with an image attachment to the selected 
recipient (any e-mail address). 

[55] It should be noted that the e-mail interface is also util ized for sending images from an Ericsson 

Communicam to other mobile devices. Corrjmuxricam images are posted from the camera to a dedicated 
server, which converts these images to an e-mail with image attachments. Proper configuration of the e- 
mail recipient address allows the user to send these images to other mobile devices. Communicam is a 
specific commercial line of cameras that can be attached to phones. It is referred here to a general 
camera attached to a phone, 

g^SMS Module 

[56] The MMR allows users to send messages in several SMS based formats. Picture Message for Nokia 
phones, and EMS messages for Ericsson phones are supported. 

[57]mcorning messages are transcoded into PM and EMS, dividing the original message into up to 6 SMS 
messages. The recipient's phone receives these SMS messages, and concatenates them. When all SMS 
messages have arrived, an application on the mobile device displays the message content. 

[58] It should be noted that MMR can also receive PM and EMS messages originating from mobile devices, 
and transfer these messages in different formats to other devices. This feature requires a special 
agreement between the SMS service provider and the MMR operator, in order to forward ail 
concatenated SMS messages through the MMR 
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[59] The conversion a foil message {pre source with source objects) is a conversion where certain 
constraints and relations between the media objects requires more processing and application of 
"business rules*: for example, if an EMS message which is 6 SMS long is sent to a Nokia phone (NSM 
messages are up to 3 SMS long) some or all of the following operations may take place: 

[60] a. The images get resized to reduce their ske in bytes, 

[61] b. The Audio files get truncated to reduce their size in bytes. 

[623 °» formatting may be removed to reduce total message size. 
gy^MMR Logic 

[63] The MMR-Logic controls the behavior of the MMR Using the MMR database, and set of configurable 
rules, the MMR Logic selects the most suitable message format for each recipient It then determines 
the correct data Qow path for each of the possible message transactions. The MMR-Logic is also 
responsible for the on-the-fly gathering of information about users and their mobile devices. This is 
performed by e.g. registering the WEB/WAP user-agent of the phone when it sends requests, or by 
identifying the type of the message sent from a phone (e.g. an EMS message) which indicates that mis 
phone can send/receive message in this format, 
— MMR Da tabase 

[64] The MMR database stores information about the system users, such as name, phone number, phone 
model etc. Message contents, i.e. images, audio and text, are also stored in the database. The MMR 
database also contains information required by the O&M block. 

[65] The MMR O&M functions provide the MMR system administrator with an array of tools to monitor 
and control the behavior of the MMR. A Web based interface, provides the administrator access to web 
pages, arranged in groups according to the functional blocks in the MMR- 

[66] The O&M also provides the administrator with a messaging page, which allows him to send messages 
in all formats to mobile devices, 
fir^MPS CHent 

[67] The MPS client translates the required transcoding action, as determined by the MMR-Logic block, 
into an XML request This request is then posted to the MPS server rack. The MPS client then parses 
the response, and extracts the transcoded image. Further details are found in section IV J 
jjj^MPS (Media Process ing Server) 
[68] The media-processing server handles die message transcoding from one format to the other. Other 
image processing functions such as face detection can also be called via the XML interface, 
■y^VASP RPC Module 

[69] The MMR provides an external interface to Value Added Service Providers (VASP), allowing remote 

invocation of MMR functions via an XML RPC interface. 
[70]Tthe XML RPC imrastructure can be easily expanded to include additional remote procedure calls. 

Details are found in sections IV H and the last section titled MPS control interface document 
■^Internal WAP GW 

[7 1] The MMR hosts an internal WAP gateway. This gateway is required to support mnctionality not yet 
supported on commercially deployed gateways. The internal WAP gateway allows the MMR to 
send/receive MMS messages, as well as use WAP push for message notifications. The inclusion of an 
internal WAP gateway is optional, not a must An external MMS compliant WAP gateway supporting 
segmentation and re-assembly (S AR) and MMS tags/mime types can be used- 
j3g-fatemal SMS GW 

[72] The internal SMS GW is used due to special functionality required for receiving EMS and PM 
messages. The SMS gateway is an interface layer/mediator for receiving and sending the SMS 
messages from/to an SMS center (SMSC) via the prevailing protocols such as UCP,SMMP,CIMD etc. 
R\G Details of Selected Functional Blocks 

■k^User Web Pages - The MM R Web Site 
[73] The public MMR main web portal contains links to at least the followingr functions: 
[74] Link to the user's personal web based "Messaging Application" 
[75] Link to the "Photo Album" 
[76] Link to the "User Registration Page" 

aWWeb-based Messaging Application 
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[77] Hie web based messaging application provides similar functional capabilities to the wap based 
application. User's may enter their personal messaging page, by using the same user name and 
password as used on their mobile phones. Once inside, user's can view and send messages in a variety 
of formats, 

[78] From the web-based application, users can also send messages with original cootenL 

fe^Web-Based Photo Album 
[79J The MMR provides a web based photo album application, allowing the user to upload and manage 
their own folders containing images and audio riles. These files can then be shared with friends, or sent 
to mobile devices in a variety of formats. 
[SO] The MMR can automatically select &e message format most suitable for the recipient, or may receive a 
request from the sender to send (he message in a specific format 
e^Wefa-BasedUser Registra tion Page 
[8 I] The user registration page allows new users to register themselves to the service. It also allows 

registered user's to update their registration information. Registration information requires the user to 
submit some personal details, as Ls accustomed in web based email services. In addition to this 
information, the user can be asked to submit information regarding the model of his mobile device. 
The B-Mail Module 

[82] The MMR allows users to send E-mail messages with image attachments to mobile devices. It also 
allows mobile users to send E-Mails with image attachments to regular e-mail addresses, mcoming E- 
mails are parsed. The e-mail subject is sent as the message text, while each of the image attachments in 
the original e-mail are transcoded for the mobile device. Depending on the amount of attachments, the 
recipient may receive several message^ and a format most suitable to his mobile device. Outgoing e- 
mail messages use SMTP to send the message text along with an image attachment to the selected 
recipient (any e-mail address). 

Bj~- Email Server Account Set-Up 
[83] The e-mail server needs to be configured to receive all mails addressed to a selected domain , e.g, 
picsAicngo.com , All mcoming e-mail messages in this format are accepted by the e-mail server^ 
Furthermore, the server is configured to create an event for each incommg message. This event triggers 
the MMR to handle the new message an described in the sections below, 
kl— Email to Mobile Device 
[84] 1. Receive messages sent to : user-phone-number@pics,ucngo.com e.g. 

97254985026@pics.ucngo.com 
[85] 2, The e-mail is handled as follows: 

[86] i) The email subject is extracted, and used as the message text. (The message text is limited to 120 

characters (configurable)). 
[87] ii)The phone is extracted from the email address and used as the recipient's number. 
[88] iii) The Image attachments are used as the message images. Each attached image generates a new 
mobile message, with the same message text The maximum allowed message size is 1 50Kbyte 
(configurable) 

[89] 3. The message is stored in the message table of the MMR database. 

[90] 4, The MMR-Logic selects the optimal message format for the recipient. 

[91] 5. A message in the selected format is sent to the recipient 
eW-Qutgoing Error Messages 

[92] Incoming e-mail messages that can not be handled according to the logic above, will generate an error 
message. This error message will be sent to the e-mail originator, to notify him that his message could 
not be handled. For several expected cases, the exact error will be given, to explain why the message 
could not be handled: 

[93] i) Recipient number is not a valid number, or is unknown to the system 

[94] ii) Message does not contain a valid image attachment 

[95] iii) Message attachments are larger than the allowed quota. 

d jh^Mobile Device to E-mail 

[96] Mobile devices may send e-mail messages via the WAP messaging portal, 

[97] During the process of sending a message, me WAP user is provided with a "send as" link, allowing him 

to select from a list of optional formats. 
[98] By selecting " send as e-mail" the user prompts the following chain of events: 
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[99] i) A new e-mail message is composed, 

[100] ii) The image original is taken from the message database, and sent as an e-mai! attachment. 
[ 101 ] iii) The message text, as edited by the user is sent as the e-mail body, 

[ 102] iv) The e-mail subject is a generic message in the form of: * You have received a new mobile e- 

mail from <sender number>" 
[103] v) The recipients e-mail address can be entered in one of two ways: 

[1 04] 1 . If the e-mail recipient is a registered user, the sender may type in the recipient's phone number, 

and the MMR will lookup the recipients e-mail address from the database, 
[105] 2, If the recipient is not registered, or if his e-mail address is not known, the sender will be directed 

to a wap page torn which he can edit the required e-mail address, 
j^SMS based Messaging 
[106] The SMS-based module is k charge of generating at least the following message format: EMS, 

PM, WAP link notification (SMS or WAP-push), WAP-push MMS notification, Text SMS for 

devices that do not support images. 
[107] Furthermore, the SMS module includes an SMS link layer, capable of receiving EMS and PM 

messages from mobile devices. The link layer can then concatenate the fragmented SMS messages that 

make up an EMS or PM and extract the message image and text. These messages can then be 

transcoded into any of the supported formats, 
[1 08] The MAS core is a group of java servlets, that handle image transcoding management / message 

transfer / database functions / billing and O&M inactions. 
[109] These servlets have external interfaces to an Email server , SMS center and WAP / WEB gateway 

allowing the MAS to interconnect between devices using these protocols. Refer to the : SMS / EMS / 

PM Module for a more detailed block diagram of the SMS / EMS / PM modules. Refer to the : MAS 

E-Mail Module for a more detailed block diagram of the POP3 / SMTP modules. 
[1 10] The PM/EMS/SMS receive will be handled by a dedicated servlet, it will interface all mconiing 

SMS's handled by the SMS GW. It will encode the booming SMS's using the following top level logic: 
[1 1 1] 1) Detect type of message single, concatenated 
[112] 2) For Single Message: 
[1 13] a) Detect Type of message Text, PM, EMS 
[1 14] b) Extract Image or Text from message 
[1 15] c) Post data to with text & phone number to sms handler servlet 
[116] 

[117] 3) For Concatenated Messsage Store in local db with message ID 

[US] 4) When Last message received- (from analyzing XX,YY,NN : XX - msg id, YY - total number 
of msgs, HN- msg sequence in the UDH) and tracckin sequence of received messages, do: 

[119] a) Concatenate message data 

[120] b) Detect Type of message Text, PM, EMS 

[121] c) Extract Image or Text from message 

[122] d) Post data to with text & phone number to sms handler servlet 
MMR-MM1 System Logic 

[123] The MMR Logic module determines the data flow path and transcoding type used on messages 
that go through the' system. Sub-section 4(a)defines the chain of events that take place, for eaclx of the 
possible combinations of input and output formats. However, there are cases where the recipients 
phone capabilities are either not fuBy known, or the recipient's phone may be able to accept messages 
in more than one format. 

[ 124] Subsection 4(b) deals with selecting the correct message type for the recipientThis sub-section 
deals with scenarios where either the sender or recipient's information is either not known to the 
system, or it conflicts with previous mformation stored in the MMR about the user 
aWTranscoding Matrix 

[125] The MMR enables messages to be sent from one device to the other, automatically transcoding the 

message content from the source device format to the target device format. 
[126] The supported formats are ; WAP / WEB / E-Mail / PM / EMS / MMS / SMS. 
[127] The following sub-sections describe some of the various transcoding actions taken for each 

combination of source and destination formats. 
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[128] Image source data is already in the database. 

[129] Send notification to the recipient with a link to (he new image, 

[BO] Transcode the image when the recipient enters the message page according to the UA used. 

[131] Upload new image via HTTP to the database. 

[132] Send notification to the recipient with a link to the new image. 

[133] Transcode the image when the recipient enters the message page according to the UA used 

[134] Upload new image via HTTP to the database.[1 35] Transcode the image to the expected 

recipient phone type. 

[136] Transcode the image from the format above to the PC monitor format. 

[137] Display resulting image wrapped in some html, 

[138] Delete the image original from the database. 

[139] Extract images and recipient numbers. 

[140] For each recipient send all the images. 

[141] Send notification to the recipient with a link to the new image. 

[142] Transcode the image when the recipient enters the message page according to the UA used. 

[143] Take image original from the database. 

[144] Transcode to GIF or JPEG to a size no larger than 25Kbyte (Configurable) 

[145] Send to recipient's e-mail address. 

[146] Take image original from the database. 

[147] Transcode to PM / EMS / MMS according to recipient phone type. 
[148] Send message. 

0^ wib to m mm i urn 

[149] Upload image from web or photo-sharing site. 

[150] Transcode to PM / EMS / MMS according to recipient phone type. 

[151] Send message. 

[152] Extract images and recipient numbers. 
[153] For each recipient send all the images. 

[154] Transcode the image to PM / EMS / MMS according to recipient's phone type. 
[155] Send message. 

#-WAM Wmi to SMS 
[156] This mode will be nsed when the recipient's phone cannot display images. 
[157] The message text will be sent as an SMS to the recipient. 

fltty-HMS IWUm WAP * 

[158] Receive and store EMS/PM as fragmented SMS messages. 

[ 159] Link fragments to a complete EMS/PM 

[ 1 60] Send notification to the recipient with a link to the new image. 

[161] Transcode the image when the recipient enters the message page according to the UA used. 

^tt£=!MS fWLto EMS f PM * 
[1 62] Receive and store EMS/PM as fragmented SMS messages. 
[ 1 63] Link fragments to a complete EMS/PM 

[164] Create and send an EMS or PM according to the recipients known device capabilities. 

QZ) EMS im fc> MMS * 

[1 65] Receive and store EMS/PM as rragmented SMS messages. 

[ 1 66] Link fragments to a complete EMS/PM 

[167] Transcode EMS/PM into MMS format, 

[168] Initiate an MMS transaction with the WAP Gateway. 
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[1 69] Slrip images from the original content 

[1 70] Send the first 1 60 characters of the text message as an SMS message. 

^H%— MMS to &U fenft&te 
[171] Receive the MMS message. Extract the content files and the SMIL 
[172] Insert the all image and audio files into the database, 

[173] Enter the SMIL file into the database, Transcode the SMIL informatm into HIMUWMUEMS 
formatting information if the targets are WBB S WAP,EMS respectively. If the target is email or an 
MMS phone that does not support SMIL, the media objects (MIME objects) may be reordered based 
on the information in the SMIL description to ensure proper viewing order between the various media 
objects, 

[174] Send the message to the selected output format, as if it came from the WEB. * - Requires 

redirection of concatenated SMS messages by the SMSC through the MMR* The MMR then handles 
EMS/PM messages according to the above logic* 

k^-Sender and Recipient Logic 

[175] In order to properly perform a message transaction, full knowledge is required about the sender 
and the recipient This is required to allow for optimal transcoding of the input message to the correct 
output format. However, there are cases where either the sender or the recipient (or both), are "not 
completely 1 ' known to the MMR. Hie data stored in the MMR database may be incomplete or 
inaccurate. For example, a user might be registered to the service, without the MMR knowing which 
device is being used. This information may also change when the user moves his SIM card from one 
device to another. In other cases, the user might not be registered in the database at all. The purpose of 
this module is to perform the correct logic decisions, to make the most out of the data that is known to 
the system. Furthermore, the sender and recipient logic are used to gather information about the system 
users in an un-formal way, by correlating mformation such as phone numbers, device user agent, and 
incoming message formats. This information is added to the information submitted by the user, during 
the registration to the service (which is not mandatory, but recommended). Further details are included 
in Section I 

c)^)n-The-Flv Data Collection 

/ Bm i MMS Comity (FMaM.BMS^ut 

[176] When an unregistered or a partially known user sends a ?M , EMS , or MMS message to an MMR. 
recipient, the MMR can register the sender on the fly. The purpose of this action is to update the 
database, and add users on the fly. If the user was already registered, the MMR checks that the user's 
capability to send messages in this format is already known. 

^OftfMag the thvt\ WAP ProSte 

[177] When a registered user sends a WAP message to an unregistered recipient, this recipient receives 
an SMS notf ftcatioiL When the recipient enters the message page, his phone's User-Agent becomes 
known to the system. At this point the MMR can add the recipient as a new user, and assign the correct 
device to him. This function is also useful for registered users who are now using a new phone model 
The database can be updated with the new user agent. Accordingly, other capability flags, such a PM 
and EMS might now change. The exact same description also applies for phones using HTTP (standard 
WEB) browsers - there too the browser specifies a User Agent 

^^^tmMt^ XZm 4m and d&vfeed&& 

[178] At any given moment, the MMR database might hold mformation about the user and his mobile 
device. Since some of the message formats may operate by using the user's capability flags alone, some 
users may not have a registered device type for extended periods. When user's register themselves 
through a dedicated registration process, or when users enter a WAP session their device becomes 
known. At this point it is important to verify that there is no discrepancy between the user's capability 
Hags, and the devices 1 capability flags. Hie synchronizing process forces the devices' capabilities on the 
user. 

d eselected Message Type Logic 
[1 79] This section explains the logic implemented in the MMR Logic Module, to select the correct 
message type for the recipient The logic is divided into a case where the recipient is registered (at 
least with partial data), or when the recipient is unknown to the MMR. 
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[180] For each transaction, this logic should bring the system to the following state: 

[181] There is a valid sender data structure. 

[182] There is a valid sender device data structure, 

[1 83] There is a valid recipient data structure. 

[1 84] There is a valid recipient device data structure. 

[185] The format of the incoming message is well defined. 

[1 86] The format of the required output message is well defined. 

[ 1 87] The flow of events required to carry out this transaction is well defined as discussed above in 4(a). 

[188] The selected message type for the recipient for any transaction is effected by the following 
parameters: 

[1 89] A direct request by the sender for a specific message type, 
[1 90] The user's device capability flags. 
[191] The user's capability flags, (if the device is not known) 
[192] The usefs preferred message type. 

[193] The device's preferred message type (if the user did not specify one) 

[194] The original message format 

[195] 

[196] The capability flags show if the user can accept message in the following formats: 

[197] MMS / LEMS / SEMS / VU I WAP Flag. 

[198] 

[199] Due to the information required before a decision can be made, the recipients data must be known. 
The receiver data may either be known because it was stored in the database, or because it was 
temporarily created for this transaction, as explained in section 4(b). In any case, at mis point there can 
no longer be a discrepancy between the user's capability flags and his device's capability flags. 

[200] Given mat the recipient is known, the selected message type will be chosen according to the 
following logic. 

[201 ] If the sender requested a specific format, that format is selected. (Forcing the format by tbe sender 
may result in the message not being sent This is not the normal mode of operation. In the normal 
mode, the sender selects "automatic" and the MMR decides the best format automatically.) 

[202] If the sender mode is automatic, the user's "preferred message type" is compared to the devices / 
user's capability flag. If it is a legal selection, the message is sent to the user in his preferred format 

[203] If the user has no specific preference, and user's device data is the next dominant information 
according to the following logic: 

[204] o If one of the optional formats of the device allows the message to be sent without being 
transcoded, that format is selected, 

[205] o If the message must be transcoded, and the device has a "preferred format*, that format is 
chosen. 

[206] o If the device data doesn't specify a "preferred format", the best of the format options is selected 

according to tbe following order: MMS, WAP, EMS, PM. 
[207] o If the user's device is not known, the user's data is the next dominant information according to the 

following logic: 

[20S] o If one of the optional formats acceptable by me user allows the message to be sent without being 

transcoded, mat format is selected, 
[209] o If the message must be transcoded, the best of the format options is selected according to the 

following order: MMS, WAP, EMS, PM, 

^Sd&tad Me&sa$fe Typ& for m mz^msi mtiptem, 

[210] If the sender requested a specific format, that format is selected. (Forcing the format by the sender 

may result in the message not being sent.) 
[211] If the sender did not request a specific format, the message will be sent according to a default 

table. The default table will be able to select a default output format for each input format This table 

will be configurable with a simple editor, Table 1 is an example of the default output format table. 
[212] A temporary recipient data structure is created with coherent information to allow the selected 

transaction to take place. 
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[213] 

[214] Input Format WAP WEB E-Mail PM EMS MMS 

[215] Default Output Format WAP WAP WAP PM EMS MMS 
[216] Table 1 : Default output format table, 

fe^Trar&rmtting pre-source iofonnation to server 

{217} The pre-souree information Is transmitted to the server. The following 
description along with the figures 9*11 referred to herein provides, to a 
salted artisan, further explanation on the transmission of pre-source 
information* 

aj^-User Control Web Page 

[218] This is a site that will all ow users to change the attributes; the system 

holds about their phone. Authentication wilt be done as fellows: 
[21§] User enters a page, 
[220] Enters his phone number* 

{211} System sends an SHS with a 4 digit code to the user while he is 
surfing^ 

{212} User will be able to change his phone type, EMS capabilities etc* 
[223] User pressed submit 

[224] System prompts the user to get the secret code from the SMS inbo& 
[225] User will enter the code and his registration details wit! be changed, 

bMSP Pages 

[226] The fallowing sections are pseudo-code descriptions of the JSP pages, 

[227] Enter your Phone number 
[22&] Enter your Password 

[229] Struct userstruet * GeWserStmc«ByPhoneNumber(number) 
[230] Bool isvatid « AuthentfcateUser(number, userstruttpassword) 
[231] If (isvaiid) Goto Main^jsp (2,3) else if (Try again ?) goto Sogin,jsp 
[232] If ("Password* (1 "New user*) Goto Fftrgotornewjsp 

[233] Enter phone number 

[234] Bool newuserok « adduser<number, user^agent) 

[235] If (newuserok) Bool passwrdsetok « SetNewPassword(number) 

[236] if (passwrdsetok) userstruefc » 

GetUser$tructByPhoneNumber(number) 
[237] Bool smssent = SendSMS(userstrucfcpassword , number) 

[238] Bool gotmsgvecfc GetUserMe^ageldVectortconst number, 
msgidvecfc) 

[239] If (gotmsgvect) numofmessages * msgidvett.stee 

[240] Set number in the brackets (i.e. Messages ) 

[241] Bool gotarehiveveet « GetArchiveNameVedt(archivevect) 

[242] Int numof&rchives - archiveveefcsifce 

[243] Set number in the brackets (i*e Archives ) 

[244] If "Messages* goto messages^sp 

[245] If "Archives* goto archives.jsp 

[246] tor (I~Q ; I « numofmessages) CreateUnkToMsg(msgidvect[l]) 
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[247] if *Hessage* pressed* goto msg.jsp (2.5) 

[2.48] if "Delete Air pressed Bool deieteOK - DeleteMIMessages(msgldvect) 
[249] if (deleteOK) goto MessagesDeleted,jsp, and then goto main.jsp 

[250] msgStruct = GetMessageStruct(msgidveet[I]) (including 

GetimageFilelD) 
[251] String msgixt = msgStruettxt 

[252] Victor < ehar> imagejsuffer * msgStru<±,GetImageBuffert ) 
[253] DevStruct SoureeDeviceStruet ■ 

GetUserOe>^ByNumber(ms9Strud:,From) 
[254] String UA «GetUserAgent$tringFromSession( ) 
[255] DeviceStruetTargetDeviceStruct = GetysemeviceBYUserAgent(UA) 
[256] XMtreq - GetXMlrequest « (SourceDevice, TargetDevice , 

imagejbuffer) 
[257] XMUmsporae - PostXMLrequest( XMLreq ) 
[258] OuUfftagOuffer = GetlmageFromXML (XHlresp) 
[259] Output message„wmi 
[260] If *$end* goto sendjsp 
[261] If *Deiete Message* Bool msgdeleteOK - 

DeleteNessage(msgStruckmsglD) 
[262] If (msgdeteteOK) goto MessageDeteted,jsp, and then goto 

messages^sp 

[263] Edit text edit box, 
[264] Edit number edit box. 

[265] If "Send as* pressed goto #sendas card. Select message type, 
[266] MsgStruct = CreateMessageStruct{From t To , type , text * 
imageFilelb) 

[267] If *Send* Bool sentDK « SendMessage(msgstruct) 
[268] If sentOK goto sentok.wml and then goto messages4sp 
[269] If ((IsentOK) m (type=wapwap) ) goto sendfeiiedjspjsp 
[270] If ((SsentOK) a& (type^wapemaii) ) goto noemaitaddressjsp 
[271] If ((SsentOK) && (type^wapemspm) ) goto notemsabtajsp 

[272] if *Entor Address* goto EnteremaiLjsp 

[273] MegStroctemaiiaddress « emaiiaddress, 
[274] sentOK = SendMessage(msgstruct) 
[275] if *fbt address* goto enterma&wml 

[276] for (1-0 ; I a numofarchives) CreateUnkToArchive(archivevect [I]) 
[277] if *Arthive" pressed, goto archive,jsp (2,10) 

[278] Bool GotVector* GetArcMessageIDVector{AirchtvelD , msglDVect) 
[279] tor (1=0 j I * nurnofmessages) CreatoLinkToMsg(msgidvect[I]) 
[280] if "Message* pressed, goto msg.jsp 

[281] if *Deiete Alt* pressed Bool deleteOR * DeteteANMessagestmsgidvect) 
[282] if (deteteOK) goto MessagesDeletedjsp, and then goto maln.jsp 
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<a— MAS-MPS client 

[283] The MPS client block enables MAS servlets to use MPS transcoding 
services, as well as supplying an API for XML and Base64 functions. Listed 
below are the main functionalities of this block* 

[284] Encoding and Decoding of -images from binary to base64 ascli* 

I 285] Creating XML transeode request, 

[286] Posting XML requests to an MPS server rack. 

[287] Receiving and analyzing XML responses, 

[288] Handling MPS errors* 

i4ZIh~Detuils of Media Processor 



[289] The MEDIA Processor provides image processing and transcoding for 
purposes of image enhancement and terminal compatibility* Na\ ve 
transcoding may result in unreadable content on the small screen of a 
mobile terminal. The Media Processor enhances the image to correct such . 
faults when the content type is identified* the MPS also supports audio, 
ringtones, animation, video see for example AudioTranscode* 

[290] Communication with the Media processor is implemented using XML 
interface* The Media processor reports success or failure for an entire 
message as well as for each individual operation of the message* The 
media processor supports processing multiple Images within a single 
message* 

[291] At least the following media processing functions are available to 

render message images for display on user"s device* 
[292] Adaptation furxettons- media format convert- from (Progressive JPEG, 

Baseline JPEG, JPEG 2000, GIF87, GIF 89A, WBMP, BMP, PNG, EMS, Nokia 

PM) to (Progressive JPEG, Baseline JPEG, JPEG 2000, GIF87, GIF 89A, 

WBMP, BMP, PNG, EMS, Nokia PM) induding colour palette adaptation, all 

based on a client submitted device type parameter* 
[293] Image content selections are provided to identify the type of image 

(e*g* - Photograph, Face, Document (e*g* FAX), cartoon. Synthetic (e*g* 

chart), Panoramic (e*g* scenery), 
[294] The MEDIA Processor includes a facility to smart compress images 

(VGA picture with smart JPEG compression takes maximum storage of 

approximately 50k)* 
[29S] The Media Processor is capable of being shared by multiple clients* 

4^-Enbancement functions 

[296] The media processor provides the following media processing image 
enhancement functions: 

[297] Brighten (dark). Darken (overexposed), Enhance, Colour balance, 
Remove Noise, threshold(local adaptive, standard), adjust levels, sharpen 
(radius, intensity, automatic), de-blur, smooth, histogram equalise, invert, 
flip(mirror), erop(erb!brary or parametric), Remove artefacts, 
resteefnearest neighbour, bi-cubic, bilinear, maximum/minimum 
neighbour, line preserving), salt and pepper removal, local illumination 
correction (arbitrary, emphasise edges), histogram equalisation, 
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histogram manipulation, Brightness, Contrast, Colour modification, Rotate 
(90, 180 or 270 degrees)* 

fe— -Auto-Enhancement functions 

[298] The media processor provides the Mowing media processing auto- 
enhancement functions: 

[2993 Auto level, auto crop, auto colour balance, auto image content type 
detection, Image classification and Optimisation Processing* Add text as 
graphic, add background, image manipulation (warping), Image framing, 
combine images* Add objects (hair, eyeglasses ete*), Include image in 
postcard / template. Camera calibration for common mobile camera 
types. 

3^—Jmaee Stitching 

[300] The media processor provides the following image Stitching; stitch 
360-degree panoramic and stitch fax* Full stitching 2 images / arbitrary- 
length series. Image pair matching, image merging, given shift 
parameters, Image stitch/match given assumptions (e.g* horizontal only), 
stitch (Brightness, Contrast, Colour)* 

4r— Advanced Functions 

[301] The media processor provides the following media processing 

advanced functions; 
[302] Detect face? detect eyes* OCR Recognition, Bar code Recognition, 

picture object recognition. Image recognition (e.g* content type 

recognition to permit optimal transcoding), 

g— — Waten n^kixig 

[303] Watermark detect and add functions shall be provided for WBW and 
JPEG images. A watermark shall support a minimum of 19 decimal digits, 
[304] 

IVSr—Idettafying display characteristics 

[305] The following code segment explains display characteristics 

identification. 
[30$] ~ <target-device> 
[307] * <ptatform> 

[308] <manufacturer>Ericsson</manufacturer> 

[309] <model>R320</model> 

[310] <ROM-revision>n/a</ROM-revision> 

[311] «User^Agent>ErtcssonR32Q/RlA</User-Agent» 

[312] </p»atfbrm> 

[313] - <network-connection> 

[314] <nc-type>GSM/CSD<:/nc-type> 

[315] «nc-speed>9600</ne-speed> 

[316] </netw^rk*connection> 

[317] - <target-display> 

[318] «hort£ontal>88</horteonta!> 

[319] <hori2ontal-scroll>88</horiEonta)-scroll> 

[320] <vertical>52</vertical> 

[321] <vertical-scroll>110«/vertical-scroil^ 

[322] «dpi>n/a«/dpi> 

[323] «pixel-ratio>1.24</pixel-ratio> 
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[324] - <eotors> 

[325] <type>B/W</type> 

[326] <number>2</humber> 

[327] <blt-Qrfsngement>r^Q</bIt-arfQngement> 

[328] <paletteHArr>rtfa«/patette^UT> 

[329] «gamma /> 

[330] <brightoess f> 

[331] </colors> 

[332] <emstype>none</emstype:> 

[333] <pdu>1300</pdu> 

[334] <maxwpdksi£e /> 

f335] «maxpbtels /> 

[336] «/terget-disp!ay> 

[337] </terget-devtee>- 

[338] 

rff^fi — Additional Processing by Media Processor. 

[339] The Media Processing Server (MPS) is designed to handle all media 
types, including formatted text, images, animations, audio and video, 
with an emphasis on advanced processing algorithms. In a nutshell, some 
of the following funcltonallties are provided i 

[340] 1, Image Tiranscode - Optimally convert content for a target phone. 
Automatically performs res&ing, color palette reduction, compression, 
rotation, watermark detection and more. The transcode operation is 
controlled by a rule based system with configurable parameters for 
bandwidth utilization, format usage. Quality of Service and content 
preferences. Performs different transcoding operations based on 
automatic detection of the content type, 

[341] 2. Audio Transcode - Similar to transcode for audio files. Useful for 
converting audio found on the Internet to MMS phones. Also supports 
conversion of ringtones between the different formate existing today. 

[342] 3, Video Transcode - similar to image transcode for video files. Also 
supports cross media conversion - video to animation, video to still image, 
video to sound track, 

[343] 4, Image manipulation package: 

[344] 4,1, Rotate - rotates an image by a specified amount with a selection 
of interpolation methods, 

[345] 4,2, Pestee - resizes an image with several Interpolation methods 
including special modes for phone screen with a small number of 
colors/non-square pixels 

[346] 4.3, Brighten - enhances the image brightness - useful for dark images 
and for adapting to phones with a nonlinear Gamma curve, 

[347] 4.4. Darken - decreases the image brightness - useful for over- 
exposed images and for adapting to phones with a nonlinear Gamma 
curve, 

[338] 4,5, Enhance - combines color and contrast enhancement of an image. 
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[349] 4*6, Color balance - performs color balancing of Images taken by low 

quality cameras or in difficult lighting conditions* 
[350] 4.7, DenoiseSpeckle - noise removal for low-light/noisy camera/data 

trasnsmisslon errors situations, 
[351] 4.8, Threshold - blnarteation of images for B/W screens. 
V$52} 4*9, Adjust levels - parametric contrast adaptation/ 
[353] 4*10* Sharpen - fast parametric correction of blurry Images. 
[354] 4*11, Deblur - special sharpening for camera images taken In low light 

conditions* 

[355] 4*12* Smooth - smooths a noisy image* 

[3561 4*13. Histogram equalize - automatic contrast range enhancement, 
[357] 4*14* Invert - performs a color/grayscale Inversion* Useful for certain 

synthetic Images on low contrast phone screens* 
t358] 4*15* Flip - fast mirroring operation* 
[359] 4,16* Crop - cut a part of the image, 
[360] 4.17, ArtifactRemove»3PEGart5factremovaUUsefulforhighly 

compressed 3PEG images (e*g, those transmitted over wireless links). 
t361] 4.18* DenoiseSaP - Salt and Pepper noise removal, 
[362] 4*19. LocHlumCorrect - Correction of lighting non-uniformity. Useful for 

images of printed text, 
[363] 4*20, PremHtetEq - advanced histogram equalization for images with 

dynamic range problems, 
[364] 4*21* ColorPaletteAdapt - Reduce the number of colors In an image 
using a fast algorithm. Useful for image file stee reduction/adaptation to 
phone screens with a small number of colors* 

[365] 4*22* FaceDetect - automatically detects a human face in a frontal 

facial image. Useful for capturing the most important part of an image for 
display on a limited size screen, 
[366] 4,23* EyeDetect- automatically detects the eyes*nose section of a 
frontal facial image. Useful for capturing the most important part of an 
image for display on a limited stee screen (e.g, Nokia Picture Message). 
[367] 4.24* Add Text- Add formatted text to an image (with font selection)* 
[368] 4.25, Add Object - Add an object (hat* eyglasess etc) to a photo, 
[369] 4*26, Add Frame - Add a frame (several selections) to a photo* 
[370] 4*27* Add Effect - artistic effects (warp, sphere, twirl etc.). 
[371] 4,28* Embed Watermark - embed a watermark In an 

tmage/audio/vldeo file* k t , 

[372] 4.29* Detect Watermark - fast detection of an existing watermark in an 

imaqe/audio/video file, 
[373] 4*30. Smart Compress - reduce the file stee of the image/audio/video 
file to below a specified limit. Useful for reducing network bandwidth and 
for overcoming memory limitation in handsets* 
[374] The MPS supports In a single product the complete range of processing 

requirements for the full spectrum of future MMSC Infrastructure users: 
[375] 1, The phone MMS user, composing and sending an MMS from a 
phone. In this scenario the primary need is for fast transcoding and 
automatic content type identification and processing. For example, images 
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taker* by a user with a camera-phone need JPEG artifact removal, 
automatic contrast and color enhancement and face/eye detection far 
maximum utilization of target display screen size, 

[376} 2. The Internet MMS user, composing advanced MMS messages from 
an internet-based interactive Multimedia Album. This user can play around 
with images and audio/video objects, add fcext/objects/frames to image, 
compose and use existing SMIL templates etc. Relevant functionality in 
the MPS includes support for interactive manipulation of images (adjust 
contrast, add formatted text, add a hat to a person in the image etc), 
efficient storage of images (smart compress), 

[377] 3. Advanced MMC scenarios, where a sequence of processing 

operations is performed on an MMS prior to sending - for example, detect 
watermark, block/report to billing system based on watermark info, 
compress audio component to reduce total MMS size while maintaining 
overall quality, convert video sequence to animations etc* 

[378j 4> Content providers - these providers have large amounts of content 
with specific, detailed processing sequences based on their 
preferences/knowledge of the content characteristics. Such providers will 
utilize the more advanced options of functions such as Transcode, 
compress, color palette adaptation, embed watermark etc 

■h-~Transcoding 

[379] The main functionality of Transcode is to convert an image so it will fit 
into a target device while maintaining the best quality possible. In order to 
fit an image to a specific device, the main considerations are; 

[380] 1, Resizing the Image until it Is smalt enough (in pixels) to fit the 

[381] 2. Reducing the imagers color/bit depths to the device capabilities, 
[382] 3. Converting the Image to the specified format - typically this format 

should be supported by the target device. 
[383] 4, Ensuring that the resulting file size does not exceed the memory 

limitations of the device 
[384] 

[385] The algorithm used by Transcode can be divided into three main 

stages, according to the above criteria, 
[386] 

aH-Stage I: Resized 

[387] 

[388] In this stage the image is resized to fit the target device. For better 
quality, other image attributes (like bit depth) are not reduced yet 
(actually they may even be enhanced). Different variant of the resizing 
algorithm are used for different contentType values. Some parameters 
that may Influence the result of this stage are; 

[389] Device dimensions, scroll-size, maximal allowed pixels, etc, 

[390] Source and target aspect ratio of pixels 

[391] The choice whether to use just the physical screen or the full scrollable 
screen - this is controlled by a configuration parameter, but overriding it 
in the XML-request is possible (useScroll), 
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[392] The option to rotate the image by 90 degrees in order to get a larger 
view (rotatetobest) - the default value allows rotation for small-screen 
devices only (cell phones vs, PDAs, PCs). This may be changed in the 
configuration or the request itself, 

id—Stage II: Color Fit 

[393] 

[394] At this stage the Imagers bit depth and color space (l>e, color to gray) 
may be reduced In order to best fit a device, (For example, a color image 
with 24 bits of data per pixel may be reduced to a grayscale Image with 2 
bits of data per pixel in order to fit a screen that has only 4 gray levels), 

[395] The image is run through a series of specially designed filters that 
maintain maximal Image quality while reducing the bit depth, 

[396] Specifying the contentType of the image can also control the behavior 
at this stage. For example, a lineart-type image is treated differently here, 
with filters that are designed to preserve as much detail as possible of 
lines and shapes, as opposed to a face/object image, in which the 
processing involves sharpening of facial features, or 'scenery* photo-type 
image, in which the main point is to preserve color and brightness 
accuracy as much as possible. 

Stage HI: Creating the output fife 

[397] When the image has reached its final siae and depth, it must be 
converted to the format requested (after making sure it is supported by 
the target device). This stage could have been straightforward, but we 
must also mate sure the file is small enough for the device's memory to 
handle. In some cases, after the file is created, It may be necessary to 
repeat the previous stages and create an even smaller image, until the file 
stee itself is small enough, 

tfh-Other stages: 

[398] Stage 0 in contentType « "document* consists of local thresholding, 
[399] Reiteration of the process with stricter limitations if the output file stee 
is too large, 

a_»-Waterniarkine 

[400] Watermarking (WM) consists of embedding hidden information within 
media files/objects, which may be used as part of a digital rights 
management system (DRM) - for billing, copyright, content-blocking etc 
The information content of the watermark in MPS is defined as a 19-digit 
numeric string, excluding 0 (i,e, l<«Wt4<1019), 

[401] Currently watermarking is supported for the formats jpeg, gif and png 
and is performed by hidden comments - for jpeg and png? these 
comments won't be visible through typical viewers. But It can also Include 
watermarking of B/W at the image level, regardless of the format. The 
typical scenario for watermark usage is through devices that do not 
normally manipulate images, but may send images previously received 
from an MPS system without tracking information, 

aV— Watermark functions: 

[402] EmbedWatermark - This function is used to embed the watermark 
(numeric string), It can be used only when the specified output format is 
one the supported WM formats. 
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[4031 DetectWaterrnark -This function detects the MPS watermark 

embedded in an image / media file. It is relevant only for WM-supported 
Input formats Note: The output of this function differs from typical MPS 
output - It is the watermark (or 'watermark not detected* message) and 
not an image. 

[4043 RemoveWatermark - This function is used to dear the watermark 
from an Image. Note; This operation may happen as a side effect of most 
methods for some formate /implementations. 

[405] It should be noted that currently the watermark functionality is 
designed mostly for demonstration and evaluation purposes. When 
integrated as part of a well-defined DRM system, watermarking 
functionality may include: 

[406) Method X + PreserveWatermark: To maintain the identification of an 
image after transcoding / basic manipulation, an alias of the following 
combination may be used - DetectWatermark ->wm; method-X; 
EmbedWatermark (wm). 

[407] Method X + ManipufeteWatermark: Another possibility is that the 
output-watermark will have a different value than the input-watermark, 
either by applying some mathematical function to it, or by the some DRM 
component that will issue a new value and maintain a log of the 
relationship between these values, 

3, — Overview of Image Processing Algorithms 

[408] The system introduces a large number of image processing algorithms 

designed for: 
[409] i. Image adaptation and manipulation 
[410] 2. Illumination correction 
[411] 3. Noise reduction 

[412] 4. Sharpening „ 
[413] This grouping of methods is for the survey convenience only. The 
methods are simple enough to allow good definition of the parameters 
Involved. Each method deals with common problems, relevant to image 
processing implementations. Still, the full collection of these methods does 
not allow dealing with complex problems, which are addressed by 
transcoding, premium and advanced packages. In complex scenarios it 
may be difficult to choose appropriate methods, for correcting the 
problem without introducing undesired side effects, which may degrade 
image quality to an undesired level. 

4s — Cotnmnn features 

»»— -Color treatment 

[4141 in the image manipulations, denolslng and sharpening functions the 
colors are treated independently. This means that each method is well 
defined for grayscale images. Thus it is applied with the same parameters 
3 times, once tor every RGB channel. There is no essential difference In 
handling the Indexed images vs. the continuous color/grayscale Images. 
Such treatment of color channels is simple and intuitive. It allows better 
understanding and description of the parameters Involved. More elaborate 
color space treatment shall be Implemented in the context of premium 
package scenarios. 
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[415] In the Illumination correction functions there are usually 2 modes of 
Implementation with separateColers parameter acting as selector. When 
set separateColors^false, the method would handle all channels In a 
combined manner. 

ttfr— Smoothing kernels 

[416] Some algorithms included (noise reduction, sharpening etc,) use linear 
convolution with a pre-defined kernel as the main processing tool. The 
most common convolution is convolution with a simple Gaussian kernel. 
However, using convolution kernels with other shapes might improve the 
performance of the algorithms, 
[417] The algorithms which are not very sensitive to the kernel parameters 
use Gaussian kernel with standard deviation automatically calculated from 
the effective radius, 
[418] The algorithms which are sensitive to the kernel parameters use one of 

the following shapes; 
[419] 1, rect - Rectangular shape is the most common, since it allows very 
fast computation. The problem with this kernel is its emphasis on diagonal 
edges, which are seldom present in the image, 
[420] 2, diamond - This is a rectangular shape rotated by 45 degrees. The 
best feature of this shape Is its ability to emphasise horizontal and vertical 
edges of man-made structures and geometrical objects. The other reason 
for emphasis on horizontal lines is the fact that they aire hardly influenced 
by discretization process, 
[421] 3, ellipse - Orcular, or more generally elliptic kernel treats in the same 
way structures of every orientation. This is a more general-purpose 
kernel, used with natural Images, 
[422] 4, softEliipse - In ellipse the edges are hard-threshold: either 1 or 0, 
In softElltpse the edges can have a value between 0 and 1, This allows a 
better approximation of disc shape. This feature is suited for linear 
operations and may cause artifacts with non-linear filters (median, 
contrast stretch etc) 
[423] 5, gauss - This stands for Gaussian filter, e,g, *gaus0707Vgaus0505* 
and *gausAuto\ The later two indexes stand far the standard deviation 
value of the Gaussian in each direction. The recommended setting 
*gausAuto* automatically calculates sigma based on the radius of effective 
coverage of the Gaussian, The Gaussian kernel allows graceful 
degradation of the pixel weight far from the center of the smoothing 
kernel. This feature is Ideal for linear convolution, 
[424] 6. M ~ E,g, W and "k2*. Reference to bank of pre-defined kernels to 
be used in special scenarios. This is a 'premium* interface and there is no 
meaning to kemelWtdth and kemelHeight 
[425] It is possible to use prolonged kernels to emphasize the horizontal and 
vertical edges. For this purpose separate parameters are defined for 
kemelWidth and kemelHeight rather than radius, 
[426] Rne-tunlng kernels for images and algorithms can be a tedious task, 
therefore some basic recommendations are given where possible, 

& — Image adaptation and m anipulation 
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{417} The elementary Image processing operations deal with Image size and 
orientation* These functions, namely Rotate, Resize, Flip and Crop, are 
available tn every basic image manipulation package* Threshold, 
Compress (with GIF output) and Invert methods are used to adapt the 
image to the display color palette and minimize the use of target device 
memory space* Resize and Threshold provide advanced modes - which 
perform more sophisticated processing* 

[428] The methods in the basic image manipulation package can be 
optimized for speed, and can include platform specific speed-ups in all 
platforms (Intel, Solaris, etc), 

[429] Xn addition to providing an image manipulation package, these 
functions enable advanced compression and building your *home-made 
ttanscode\ Typical variations of such experimentations with transcoding 
consist of a sequence such as crop->restee~>eompress, or crop-> 
(optional) rotote«> resizes quantization/threshold - to adapt large 
images to small displays* Crop allows spending all of the limited screen 
resources on the main object} resize minimizes the information contained 
in the picture and compression/quantization discards less-important 
information* Rotation is used to use the display dimensions and aspect 
ratio as best as possible* On some media the image has to be inversed 
prior to display (e*g* scanned negatives)* 

a}— -Color palette adaptation 

[430] ColorPaletteAdapt fits the image to a limited palette* This Is useful 
either when the device or file-format has a limited color capability or when 
file size Is an issue* Once the palette is defined, each pixel Is assigned a 
value from it; this is done either by assigning each pixel the nearest 
value, or dithering - a method which increases color resolution at expense 
of spatial resolution. As default, dither is used when the specified number 
of output, colors is small, but the user may explicitly specify whether 
dithering should be used* Dithering is not recommended when output file 
size is a major issue, but ts recommended when the device color capability 
is the issue. Note that CoiorPaietteAdapt with patetteName=*B/W* is 
equivalent to threshold with the default parameter, for B/W adaptation 
one may prefer using advanced modes of Threshold* 

k)— Threshold 

[431] Threshold converts a grayscale image into a discrete B/W image. It 
may used as part of other more complex conversion operations (e*g* 
Transcode), and can serve for artistic effects or image combination 
effects. For example; converting a formatted text/textured text image into 
B/W before sending to a B/W screen, reducing the color content of a 
single layer (e*g* object for combining in an image) so it will not add to 
the color palette of an image etc* It allows explicitly controlling the 
threshold level, automatically finding the optimal global threshold, or 
applying a local threshold (mode locafV2 Is usually Inferior to local). 

gj — Compress 

[432] The method compress attempts to reduce file size without changing 
the image size. It may achieve this goal by more efficient coding, reducing 
colors and losing some details. In some case it may follow an operation 
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Intended to reduce the image size. It is wise to apply compress In 
combination with an efficient Image format (l,e, jpeg / jpegSOOO for 
storage* glf for most devices) 

[4331 This function has a different implementation according to the 
requested output type. The problem of file size is important in several 
aspects: Most of the currently available phones / hand-held devices have 
limited capacity, both for single file and in total. Bandwidth in cellular 
networks may also be an issue, tn addition, in messaging system storing 
many millions of MMS messages, the typical file size becomes an issue to 
consider too. On the other hand most images entering the system may 
not be limited to the fow-Kbytes range suitable for hand-held devices* 

[434] The palette and processing power limitations of currently available 
hand-held devices makes the compression utility especially relevant for 
image/gif output mime type, 

[4351 In this case compress activates adaptive quantization procedure, which 
provides for a clear image with minimally reduced color palette. Detail 
reduction, image resting and cropping are not supported by the compress 
method and require dedicated requests, 

[437] The implementation of the algorithm is based on adaptive reduction of 
the color palette and smoothing for GIF/PNG images, and on 3PEG OCT 
quantization table variation and smoothing for JPEG images. The 
parameters are changed iteratively until the maximum quality setting with 
a file size under the limit Is reached, 

[438] It is important to know that if the target file size is too small, the 
algorithm will return an error message. This reflects the fact that even 
under a color reduction to a bltonal image the file size under the given 
compression method was too large. In this case the image should first be 
resisted, then compressed, 

[4391 

[440] The parameter available for this function is maxStee, which is the 
maximal allowed Image size in bytes. For a QF sized .image, to be 
displayed on a typical hand-held device, some recommended sizes are: 

[441] 

[442] Image /Device type MaxSlze (bytes). 

[443] VGA image ~ storage Up-to 50000 

[444] Uneart/MMS message 4000 

[445] Uneart/WAP phone 1400-2000 

[446] TS8/WAP dient 2300 

[447] Natural image/PDA with MMS/email client 8000 

[448] 

^—Invert 

[4491 Some image sources (e,g, scanned negatives) and output devices 
require image inversion. The inversion is performed for each color channel 
separately, so that yellow is transformed to blue, white to black etc. This 
is a simple function so it does not require any parameters. It Is most 
useful for e,g. synthetic images displayed on low contrast screens, 

£ — Rotate 
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[450] This is a standard Implementation of image rotations The parameter 
specifying the amount of rotation counter-clockwise in degrees Is 
mandatory Values out of 0-360 range are corrected by the algorithms, so 
that -90, 2:70 and 630 degrees rotation have the same effects 

[451] For angles other than 0,90,180,270 the output pixels of the original 
Image are not a rectangular image, this poses a choice of what 
rectangular Image should be returned and what should be the values of 
the pixels not present in the input image (in 90,270 the stee changes - 
width o> height - but there Is no dilemma defining the output imaged 
The parameter mode determines the output stee; mode=full returns the 
bounding block Image of the rotated input image, mode=crop crops the 
rotated image to the size of the original image (with the same center)* 
The portion of the output image, not present in the Input Is always padded 
with a black background. For rotation by multiples of 90deg both modes 
provide the same output 

[452] The interpolate parameter allows to choose the interpolation 
technique* tnterpoiate=bilinear Is usually a good choices It Is 
computationally efficient and does not introduce large artifacts* Selecting 
tnterpoiate=blcubic provides for a more sharp and accurate image at cost 
of computational efficiency and ringing artefacts, tnterpolate^nearest 
selection Is most efficient computationally and does not alter image 
palette. However ft may introduce some aliasing artifacts. This parameter 
Is also ignored for multiples of 9Gdeg* 

[453] This speed-optfmteed function serves to change the image stee* it can 
be used to fit an Image into a small phone screen, or to reduce an Image 
stee prior to compression and storages For example, an incoming 3 mega- 
pixel image from a high-duality digital camera may be resteed to VGA 
(640 by 480) stee prior to JPEG compression and storage, in order to 
reduce storage space requirements, 

[454] The mode parameter selects the interpolation algorithm. Beside the 
usual bilinear, bicubic and nearest methods, proprietary methods are 
supported to provide for optimal performance with various image types 
and target devices, 

g) — Flip 

[455] Flip the image horizontally (vertical«false) or vertically (vertical-true). 
For an upside-down one should call this method twice, each with a 
different parameter value (or call Rotate 180 degrees), 

lifr—Crop 

[456] Crop may be used when the final image size is limited and the more 
significant details are concentrated in a limited region of the image. 
Cropping most of the background allows applying a more moderate restee. 
After the application of crop method a rectangular area Y cut* from the 
original image Is returned. Crop's interface is the following 
[457] top -the upper bound of the image, range, 1-ImageHelght 
[458] bottom -the bottom bound of the image, top-lmageHelght 
[459] left -the left bound of the image, 1-lmageWidth 
[460] right -the right bound of the image: left-XmageHeight 
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[461] The coordinates start from the top-left comer of the Image with 
coordinate (1,1), rather than (0,0) used In some other commercial 
software, The rectangle specified by the four parameters has to have a 
positive area, so left right, top<«bottom. The edges are Included in 
the rectangle, 

fe— 'Illumination correction and col or manipulations 

[462] The illumination correction is one of the more difficult problems in 
Image processing. There are many ways to correct for improper 
illumination/ detector problems. Basic solutions work only on a smalt 
range of imaging situations. The methods given below are just the most 
simple and intuitive tools, while the premium package contains more 
complex and elaborate algorithms to deal with the problem. 

[463] The AutoLevel method with empty radius parameter is perhaps the 
most powerful algorithm available in the basic package. 

a»— Darken 

[464] This method darkens the image, and has two modes: 

[465] If intensity Is not specified, the function performs a darkening which 

may be described as the dark half of AutoLevel. 
[466] If intensity is set, the image is darkened by the specified amount 
(intensity i [range: 0-1 (0- do nothing, 1-maximal)]). When intensity is 1, ■ 
alt mid levels become black and only colors white (or brightest level in 
channel X) remains as It was. Darken with intensity X Is equivalent to 
Adjust levels with contrast**©, brightness =* -X. 

b»-Brighten 

[467] This method brightens the image, and has two modes: 

[468] If intensity is not specified, tiie function performs a brightening which 

may be described as the brighthaif of AutoLevel. 
[469] If intensity is set, the Image Is brightened by the specified amount 
(intensity : frange: 0-1 (0- do nothing, l-maxlmal)]). Brighten with 
Intensity X is equivalent to Adjust levels with contrasts, brightness * X. 

€fr=Adi ustLevels 

[470] This method gives the user full control and responsibility of the output. 

Both parameters are mandatory. 
[47i] brightness : [range: -1 - +1 (-1 = black, +1 « white), recommended 

range: -.3 - +.3 ] 

[472] contrast : [range: -1 - +1: (-l=monotonic image, +l=pure color 
Image ), recommended range: -.3 - +,3] 

[473] For positive contrast values, first the brightness is adjusted and then 
the contrast. For negative values, contrast is adjusted first. The effects of 
both these operations is accumulated, so the maximal effect when the 
contrast is positive is the sum of the contrast value and the absolute value 
of brightness. This sum should not exceed 1, or be too dose to it (i.e. 
contrast=brtghtness-0.6 will result in a white image, just like 
brightness«l), 

dfc=AtttoLevel 

[474] This is the primary function for illumination correction included in the 
basic package. The global AutoLevel (empty radius parameter) maximizes 
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image contrast, discarding the outliers of very high and very low intensity. 
This is especially useful tor Images taken in the haze* ram etc* 

[4753 'the local AutoLevel Is activated, setting some positive radius 
parameter. The recommended radius values are in range [10-30], For 
small radius values the image appears grainy. Unlike the global AutoLevel, 
the local AutoLevel stretches the contrast w/o outlier detection. This effect 
is achieved if used after DenoiseSaP, 

[476] The separateColors«true setting allows illumination/color correction as 
well as luminance correction. 

[477] 

« fr — CoiorBalance 

[478] This function tries to produce truly colorful images* by increasing die 
contrast in hue domain. Thus a white object on a blue background will 
appear yellow. This method is similar to the correct illuminant effect in 
vision. On most natural images the effect Is very smaiL Settfng mode=l is 
mandatory. The level can vary in range [0-1] with default 0,5, Both 
parameters are mandatory, 

ry~~WhiteBalance 

[479] This function calculates the mean value of the gray pfeels in each 
channel and brings it to 0,5, The pixel is reported as gray if its color in all 
3 channels obeys abs(pkeLvalue~ 0,5)«to!erance* The tolerance is a 
user-supplied parameter with default value 0,15, The color correction is a 
gamma correction, with automatically calculated parameters for each 
channel, 

gt^ColorVariations 

[480] This is a two-stage function. In the first stage the gamma of each color 
channel (e,g, red, green, blue) is changed according to intensity value 
between -1 (remove color) to +1 (saturate color), In the second stage the 
saturation of the color is changed by changing the Eudidian distance 
between each channel value and gray Image value, such that-1 stands 
for grayscale and 1 stands for saturated colors, 

[481] For a simple cmos based camera such as the Communfeam it is usually 
recommended to put 

[482] red - ,2 

[483] green = ,2 

[484] blue » >,1 

[48S] saturation * ,3 

t»— HistEgualize 

[486] This method performs Histogram equalization (no parameters). The 
resulting image has a uniform histogram (as much as possible considering 
the input color distribution). This is a common soluOon illumination 
correction, but it has side effects, such as eliminating the real color 
distribution of the image (e,g, adaptive thresholding of the result of 
histogram equalization, is likely to have poor results), 

ft— PremHtstEg 

[487] Unlike the HistEqualtee, PremHistEq trades off the speed and simplicity 
for the flexibility of operation, It has a large set of parameters and modes 
of operation which have different effects. 
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[488] P-law histogram equalization allows a trade-off between simple 
histogram equalization (pvaNO), no effect at all (pvah=l), dominant 
modes emphasize (pval»l) and dominant modes destruction (pvaKl), 
The recommended values are 

[489] Pval Scenario 

[490] 0.3 Histogram equalization of moderate intensity 
[491] 0 J Histogram equalization of small intensity 
[492] 1*5 Postorization 

[493] -0*2 Histogram equalization of very large intensity 

[494] Local histogram equalization allows histogram equalization on blocks of 

limited size* The recommended size of the block is in the range 0*2 - 0*6 

of the image size* 

[495] Number of histogram bins has quantization effect* For pvaKO it is 
recommended to use at least 32-64 bins* The algorithm uses linear 
interpolation between bins* 

& — LocIllumCorrcct 

[49$] This method performs local illumination correction and has a large 
amount of sub-methods chosen by correctionType* Other methods which 
locally correct the illumination level are Autotevel (local) and, for binary 
output, the local mode of Threshold* This procedure is effective for non- 
uniformly lighted handwritten and printed text as preprocessing to 
advanced applications, such as OCR and feature detection, but it may 
sometimes degrade the visual quality of the image as perceived by 
humans* Some safety mechanisms were introduced to limit the visual 
degradation of the image* One of this mechanism is setting 
separateColors=false to preserve the original hue of the image* 

[497] The LocMlumCorrect can produce unexpected results with images with 
very limited color palette (16 colors and less)* 

id—GIobUluntCorrect 

[498] The gtobal/block-wise illumination correction allows automatic 
correction of image curves with the following sub-methods chosen by 

[499] gamma - gamma correction, so that the mean value of the image is *5 



[S00] curve - a variation of gamma correction with highlights and shadows 
subjected to separate gamma values, 

[501] contrast - synonym for Autotevel* The addition functionality is bloc- 
wise processing* 

[502] histEq - synonym for PremHistEq with different interface (number of 
bins and power is selected automatically)* 

[5031 The correction is global, unless blkHeight and blkWidth are both set 
The recommended block size is 64x64 or 128x128* The blocks overlay, so 
that their borders are virtually invisible for block size larger than 32x32, 
The separateColors parameter allows to select color channel treatment* 
Usually separateColors=true provides for desired color correction* 
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[S04] The GtoblllumCorreet can produce unexpected results with non- 
photographic Images (e.g. llneart) and images with very limited color 
palette (16 colors and less). 

Enhance 

tSOS] This function performs a selected combination of methods based on 

enhaneeType parameter, 
[506] If the enhaneeType is empty the function performs mild color balance 

+ contrast enhancement. It can be used safely on various Input images 

and should Improve many of them. The effect Is similar to ColorBalance 

followed by AufcaLevel (global), 
[SO?] If enhane^Type^eommunicam, batch processing Is executed using; 
[SOS] o ArtifaetRemove ~> Deblur ~> PremHistEq •> Crop **> 

ColorVariations GloblllumCorreet This processing Is essentially 

acceptable for various JPEG taput images, but most suitable for 

communcam output images, 

3-. — -Noise reduction 

[509] The two most common types of noise treated by the basic package 
are*, 

tS10] t. White Gaussian noise / speckle noise 
[511] 2. Salt and Pepper (S&P) noise 

[512] White Gaussian noise appears as an intrinsic part of the cheap camera 
detectors* especially in low Illumination conditions - it is Inaccuracy in the 
pkel values - for many pixels. This is the most common type of neise, 
which appears on most of the images, 

[S13] The S&P noise, is a small number of pixels having big "errors* in their 
intensity levels. It appears as a result of interlacing/aliasing In the 
detector, faulty detector, sharpening of degraded images, communication 
problems, poor JPEG compression, scanning of analog photos. This type of 
noise Is more rare and easier to treat than the Gaussian noise, 

[514] Since the noise appears independently in each color channel, the noise 
reduction procedures are independently applied to the color channels, 

[SIS] The noise types defined in this procedure do not have clear meaning in 
line-art images, therefore a natural image source is assumed. For indexed 
Images a rich color palette is required, 

-Smoothing 

[516] The most common and simple way to deal with noisy, over-sharpened 
images and compression artifacts is smoothing. Smoothing Is performed 
by a simple procedure of convolution between the original image and a 
Gaussian kernel. k 

[S17] The output of this method is a smooth image, where the degree of 
smoothness increases with the optional intensity and radius parameters, 

[518] Since there is no common scenario for the use of smoothing method, 
there are no dear recommendations about the smoothing parameters. For 
medium smoothing which does not degrade image significantly we can 
recommend intensity =0.5, radius=3, 

b>— DenoiseSpeckle 

[519] This is essentially an empiric Wiener filter. It essentially uses algorithm 
developed by Lee (1980) with minor modifications. The main modifications 
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include a bank of smoothing kernels, the overfilter parameter and 
regulartzation process, 
[5201 This de-noising procedure is essentially a focal adaptive smoothing 
procedure, where the overfitter parameter plays the rote of smoothing 
intensity, and the smoothing kernel selection piays the role of radius In 
the smoothing method* 

DenoiseSaP 

tsnj This Is a standard 2-stage procedure of outlier detection followed by 
their replacement with local median. Changing the threshold influences 
the detection rate. Increasing the threshold will allow under-smoothing, 
while decreasing the threshold will allow over-smoothing. It is 
recommended to work with kernels of Q~25 pixels. 

fe — Sharpening 

l$2%\ On most amateur photos the image is not sharp enough. The common 
reasons are; bad focus, motion blur, JPEG compression. To increase the 
sharpness of the image one can use a sharpening procedure, 

at — Sharpen 

[5231 This function implements a standard unsharp masking procedure 
(which is equivalent to smoothing with negative intensity) if edges-false. 
In this setting the function increases the noise in the Image, which Is 
usually not recommended with originally noisy images. Setting 
edges-true will result in sharpening only over the edges, which is a 
preferable mode of operation. In this mode the radius parameter is 
ignored. The level of sharpening can be controlled by the intensity 
parameter, 

a»™ Artif actRemove 

[524) This function deals with heavy JPEG artifacts only. The artifacts are 
listed below: 

rs25) 1. Blocking: various sharp borders between 8x8 tiles 

l$2&] 2. Ringing: high frequency moif* noise in 8x8 tiles 

[Sill 3. Color spiW: color channels not coinciding with luminance channel 

[528] 4. Blur: high frequency edge blur 

rS291 The Artifectftemove method with deJPGtype^detatl settings does not 
degrade the image visual quality. This is a general-purpose tool, which 
can be used for any JPEG input However, for any specific problem it is 
recommended to use the dedicated method instead, 

h&— Deblur 

rS311 Most of the digital photos taken by tow-quality cameras appear to be 
out of focus or blurred. Simple sharpening does not solve this problem 
completely, especially in the noisy environment. Some off-the-shelf 
products (e,g, Extensts intellehance) suggest blind deconvolutton with 
adaptively selected kernel. The kernel is selected, so that maximal 
sharpening is achieved with minimal ringing artifact. While this method is 
implemented debturType=defocus, it is not recommended in most cases. 
In the presence of noise and JPEG artifacts, the proprietary 
deblurType*=premlum$harpen method proves to be more consistent and 
does not degrade image visual quality. The main difference between 
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Sharpen and Deblur (premiumSharpen) is the use multiple kernels and 
homogenteation as a part of sharpening procedure. For more sharpness 
one can use deblurType=premiumSharpenMore,. 

AO—Functional description of AttdioTranscode 

[532] The support of multiple input and output format, optimized for the 
specific device, is as important in sound processing as it is in image 
processing. The main sources of audio are MP3 and WAV files, available 
on most PCs, disks and internet sites. These formate allow very rich sound 
at the expense of the disk space. The transmission bandwidth and 
memory of the hand-held devices is very limited. The audio capabilities of 
these devices, dont always make use of the richness of the input formats 
(e.g. stereo) More efficient compression techniques (e.g. GSMAMP) have 
been defined in the context of MMS for playing/recording audio on these 
devices. The audio transcoding process performs the following tasks; 

[533] 1. Down-samples and compresses rich audio files (MP3,WAV =5> 
GSMAMP) 

[534] 2. Masks compression artifacts and allows to play compressed files on 

PC (<3SMAMR«>WAV, effect«mask) 
[535] 3. Converts GSMAMR files from abundant to minimal compression 

rates and vice-versa (GSMAMP* >GSM AMR) 
[536] 4. Supports Unix audio standards (AU to anything and anything to AU) 
[537] 5, Adds effects to audio files (via the effect string) for audio quality 

improvement without issuing a separate request. 
[538] 6. Automatically selects the conversion method via the mime type. 
[539] 7, Performs ringtone conversion between Holda Rtngtones (part of the 

NMS standard) and iMelodies, TDD polyphonic tones (part of EMS 

standard and extensions to it respectively). 

PHeGr^-Other specific enhancements 

■t^Using the MMS Box as a WAP Site 

[540] WAP terminals have a built-in WAP browser. It is possible to go 
to a Web site with the terminal, and call down relevant information. The 
server will process the information called to optimize it for display on the 
terminal, and the processed information will then be transmitted to the 
terminal for display. This information may or may not be processed 
further by the terminal or by the server, according the user's request. 
Information which has been processed (either once or twice) may then be 
stored, in the terminal, or at the server, or at another information storage 
place specified by the user. Transmission to and from the terminal may 
be by wireless or wireline communication. 

% — Converting a WBMP Into a Picture Message 

[S41J WBMP is the WAP protocol for graphics. Images on the terminal 
may be displayed in PM format, not WBMP. The server may receive a 
WBMP image, convert it into the PM format, and transmit the message for 
display on the terminal. This conversion is new because the protocols 
WBMP and PM are both new, and therefore the conversion has not been 
performed previously. 

[5423 Converting a WMBP Image into an EMS Picture Message 
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[5433 Thts innovation Is exactly the same as converting a WBMP into a 
picture message described above, except that the target format for 
display is EMS rather than PM* It will be appreciated that images may be 
converted Into any number of picture formats* be it PM, EMS, or a 
different picture format to be devised in the future, such as Smart 
Messaging* The algorithms in the server make this transcoding possible. 
Again, this is new because the protocols, WBMP and EMS, are both new, 
and therefore the conversion has not been performed previously. 

— Human Face Recognition and Display 

£544] Refer to Figure 1 attached to and incorporated into this 
application* In the picture, the woman's face may be recognized by 
algorithms defined in prior art. The invention includes innovative 
algorithms resident in the server which allow the server to process the 
relevant part of the picture, in this case the womatfs face, for display on 
the terminal* There are three types of algorithms* The first is orientation* 
The face is oriented vertically, which means that the vertical dimension of 
the relevant part of the picture is greater than the horizontal orientation. 
Some terminals have display screens that are wider than they are tall* To 
capture the full image on one screen would reqire a reorientation of the 
woman's face from vertiel to horizontal* The server knows the display 
characteristics of the terminal, and will perform thts orientation* 

[S4S3 The second kind of algorithm is "resizing** Terminal displays 
are generally smaller, often much smaller, than tine source image* The 
server will know these characteristics, and will accordingly resize the 
picture for display on the terminal* 

[546] The third group of algorithms is those which will reproduce the image 
on the terminal's display, while maintaining the integrity and quality of the 
image as much as possible* The need for these algorithms arises from 
the smalt display screen, or from the inherently lower resolution of the 
terminal display, or from other reasons* The server will know the 
characteristics of the terminal display, and will apply the correct 
algorithms for masdmum effect* Examples of such algorithms include 
enhancement, dithering, and histogram correction* 

[547] The application of any or all of these algorithms to handset displays is innovative. The 

use of prior art face recognition as part of llie system and method described herein is also innovative, 
a) — Face Binarization 

[548] FIG.18 shoes a block diagram explaining the procedure, 

[549] Image is more or less frontal, eyes should be visible, and illumination variations should not be too 
extreme. Constraints are set both by face detection requirements and by binarization requirements. Size 
of face in image should be sufficiendy big. 

[550] a. Face detection, eyes detection: 

[551] Preferably, this would be a face detection SDK (e,g, trueFace, Facelt), 

[552] b. Illumination correction (to the extent possible). 

[553] c. Facial features emphasizing: 

[554] Increasing contrast - brightening the skin, darkening features (eyes, lips, eyebrows, etc.) and hair. 
[555] Sharpening. 



32 



WO 03/001770 



PCT/EB02/04148 



[556] d. Optimal resizing - undo the blur. 
[557] e. Binarization with dithering. 

[558] Output: The eyes strip. 

[559] Additional components; eye detection. 

[560] This is especially useful when one has to display a part of the human face on screens that dictate a 
wide and short frame size - e.g. many phones have an aspect ratio of 2: 1 or more in the width/height of the 
display. In addition, the PM (picture message) format of Nokia Smart messaging dictates that images are at 
most 72 pixels wide by 28 pixels high. See examples in attached figure 26 showing the extraction of the eye 
region from an image and then converting it to a picture message. 

4 — Combination of Histogram Correction and Dithering 

[SSI] A histogram In the current context is the process by which the 
various pixel values in a grey level image are distributed on a frequency 
chart, from pure white through various shades of grey to pure black. 
Histogram correction Is the process by which some of these values, but 
not all, are lightened or darkened, but even those values affected are 
changed to different degrees. Dithering is the translation of grey level 
images to black & white by the correct combination of the black and white 
pixels to simulate grey In the eye of the user. The use of dithering for 
small screens, such as those on the terminal displays discussed here, is 
entirely new. Histogram correction, even for small screens, is not new. 
However, the use of histogram correction in the method and system 
described herein is new. Further, there is a technique by which histogram 
correction Is applied first, and then dithering, to tiranscode a very high 
quality image onto a terminal display that protrays only black & white 
images. This technique is entirely new, and is part of this invention, 

§?=^ombination of Floyd Steinberg Dithering and Random Permutation 

[562] Floyd Steinberg dithering is a well-known dithering algorithm in 
which error diffusion methods are used to create visually appealing 
dithering with relatively few fixed repeating patterns, Random 
permutation is a technique by which a few random black pixels are 
changed to white and a few random white pixels are changed to black, 
Random permutation is used to avoid *periodieity% which is a situation in 
which there are appear to be very dramatic changes in shading from one 
part of a picture to an adjacent part of the picture. This problem is 
particularly prevalent when large pictures are compressed into a smaller 
area such as a small display terminal. Random permutation softens the 
effect of these changes. Floyd Steinberg dithering is part of prior art, as 
is random permutation. However, the combination of first Floyd Steinberg 
dithering and then random permutation is new, and Is part of this 
invention. Further, the addition of this combination followed by 
transcoding to WBMP, EMS, or PM, as the case may be the particular 
target terminal, is entirely new, and Is part of this invention, 

fe— -Correction for Non-Square Pixels in the Target Display 

t563] Not all terminals have square pixel displays; sometimes the 
pixels are rectangular. If that is the case, then the server may need to 
convert say a 100 x 100 square pixel picture into say 80 x 100 rectangular 
pixel display on the target terminal. The server will know the chacteristics 
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of the display terminal, and will perform the required correction; that 
technique is new. Further, the addition of this technique with other 
techniques described here, and the transmission of the processed 
information, is entirely new, and is part of the method and system 
described herein. 

^-Traascoding of Color and S hade b Images 

rsS41 Images generally in one of three types of color or shading, 
which are are color (green, blue, and red, or other permutations), grey 
level, and black & white* Terminal displays today are generally black & 
white only, although grey level is being introduced, color displays are 
planned for the future. The server will know the characteristics of the 
display terminal, and will be able to transcode source images into a 
display format suitable for that terminal. Note that it is possible to 
transcode down, or to transcode up from white & black to grey level, but 
it is not possible to transcode into colon That is, the various possible 
permutations are color to grey level, color to black & white, grey level to 
black & white, or black & white to grey level. However, grey level or black 
& white to color is not possible. Also, needless to say, it Is possible to 
transcode at the same level* such as black & white to black & white, 
although different algorithms will be employed to to maximize Integrity 
and quality of the image on the terminal display. 

&= — Panon ""« Tma^in g on a Terminal 

[5553 Refer to Figure 2, attached to and incorporated into this 
application. Panorama imaging requires that multiple pictures of a scene 
be taken, and then various images be "stitched* together in order to 
create one continuous scene, {Hotex Panorama imaging, and the various 
algorithms employed to make it possible, is the subject of a separate 
patent application by the current assignee, UCnGo, Inc.] By the nature of 
the limited size of the terminal display, the entire picture cannot be 
displayed on one screen. Scrolling is required. In add-on, re-orientation 
of the image may also be required, as demonstrated in Figure 2, It 
should be noted that long text messages, which may not be displayed on 
one screen, may also be formatted for scrolling, and again the scrolling 
may be either vertical or horizontal, depending on the type of terminal 
display and the nature of the text. 

Sh— Embedding a WAP Link in an SMS Message 

rseei This in itself is not new, since it is part of the prior art. However, 
It is new as part of the method and system described herein, particularly 
where the SMS link message serves as a method to deliver multimedia 
content to the user of a WAP phone. 

jfe-OTA Bookmarks and Enrollment 

TS67] "OTA* means *Over the Alr\ and is a short expression for real 
time action, in this case via a radio system. To do personalised 
bookmarks and home pages OTA is part of the prior art. Each user is 
assigned a specific URL ("Uniform Resource Locator*), with a common 
beginning string and some additional user-specific string (e.g., 
http ;^wapvUCngo.com/mmsservice/userboxes/userid=45FCgEO, where 
the bold part Is user-specific part). In this way, different users receive 
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different bookmarks (URLs), and when a user operates a specific WAP 
browser, the UCnGo server receives an HTTP request with a URL unique to 
the given user, and the server will know what information to display based 
on the specific URL To give each user his or her own URL, is also part of 
the prior art, but oniy in a wireline environment* To give a URL to a 
wireless subscriber is new. To combine ULR with OTA for personalised 
bookmarks, personalized home pages, URL links, enrollment in various 
services, and other services, is new, and is part of this invention. This 
combination requires both the application of database technology and the 
algorithms defined in this application, 

M^Correctmg for Inversion of Pixels on Target Display 

[568] In some terminals, the display has been Intentionally inverted. 
That is, if a 0 bit Is usually white, and a 1 bit usually black, in an Inverted 
terminal the 0 bit appears as black and the 1 bit appears as white. 
Therefore, if any MMS message at all is sent, the display in an inverted 
terminal will show the message as a negative of the original. ATimeport 
phone, by Motorola, Inc., Is an example of such an Inverted display. This 
inversion Is not a fundamental problem, as long as the server knows the 
terminal characteristics, and can correct for the inversion. The server in 
this application does know the terminal characteristics, and will correct for 
the inversion before the message is transmitted, 

4 A— Transcoding of Text or Numerics into a Picture 

[569] The Invention will transcode text or numbers into a picture, in 
WBMP, m, or EMS format, as required by the target display. This is new, 

jA^Use of Modules Such as OCR and ICR to Identity and Process Text and Images 

[5703 The server uses OCR and ICR (Intelligent Character Recognition) 
to identify which parts of an image are text. Reference Is now made to 
Figure 3, incorporated into this application. The first step in processing an 
image Is the processing of the image into parcels such as text and 
drawing. Different processing techniques are then applied to each type of 
parcel. Rules will be applied, such as "Ignore grey level information* 
because the image may be in black & white, or "Maintain line solidity*. 
Without the parsing and application of techniques, the image will be 
reproduced on the terminal in a manner similar to what Is written as 
*Na) ve Transcoding* on Figure 3. With the invention, the "Optimal 
Adaptation* level is achieved. This process is part of the invention, 

44y-Flexibilitv Resizing 

[571} As a specific case, "flexible resisting* is a technique by which 
different parcels are resized differently; for example, text may be resized 
as little as possible to maintain legibility, whereas an image, such as that 
portrayed in Figure 3, may have a greater degree of reduction, since only 
reeognizability, not legibility, is required. Flexible resizing Is also part of 
the invention, 

■tSr-Adaptive Classification Engine for Smart Resizing 

[572] A variation of flexible resizing Is where the decision of flexible 
resisting is generated not solely by recognition algorithms, but rather by 
recognition algorithms in combination with parsed samples. The first step 
of the procedure is that various sample images are fed into the server's 
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database These Images have already been parsed, and individual parcels 
have been identified as requiring different processing algorithms, in 
various orders of operation. The parsing, algorithms, and order of 
operation for the algorithms, have been tested by both theory and trial & 
error and have found to produce optimal results. When a new image is 
then received by the server, the image can be parsed, the parsed parcels 
will then be compared to the database of parsed parcels, and the 
classification engine will then choose, on the basis J f ^^^^ 
taroet image, which algorithms and which combination of algorithms to 
apply to each parcel. This classification is "adaptive* in that in changes 
either with the addition of samples, and/or with feedback from the results 
of processing on real images. The adaptive classification engine is like a 
learning machine that applies rules and improves its own P^?f «f 
over time. The concept of a learning machine by itself is prior art. An 
adaptive classification engine for smart resizing of MMS messages is 
entirely new, and is part of this invention. 

l£— Vi-rJnrizinfl and Processing C harts and Cartoons 

T5731 The processing of charts and cartoons is similar to the 
description of Use of Modules Such as OCR and ICR to Identify and 
Process Text and Images (see above). It Is portrayed In Figure 4, hereby 
incorporated into the application. That is, the image will be P^ to 
determine the various parcels to be processed. Then rules wiH be applied. 
For display of a standard sire chart on the small display of a terminal, tor 
example, superfluous information such as a series of values on the axes 
will be removed. Also, a rule such as "remove horizontals* may be 
applied, since the addition of the horizontals to the small display screen 
will make the graph almost unreadable. For the graph itself, a rule such 
as "maintain line solidity* may be applied. The entire image will then be 
resized to the smalt display, Vectorizing algorithms are prior art. For 
example, Adobe Streamline Software uses this technique. The technique 
has not been applied to small screens such as the terminals discussed 
herein, and that is new. The combination of that technique with resizing 
and the other operations described herein are part of the method and 
system of this invention. 
f5741 

[575] This is a further explanation of the vectorizing and processing of 
charts; 



rS76] Standard charts -^graphs of a continuous onenlimensionai function, 
e g, stock charts and other temporal variables. (A bettor solution can be 
provided more easily tor a known source with known format). 



TS771 a. Identification and separation of graphics to content layers; graph, 
arid and boundaries, graph label, range text (numbers on both scales), 
background, b, gnore clutter (background, grid, possibly gnd 
boundaries). 
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[578] c. Handle range; Ignore most values; maintain mtn* max at required 
size, 

[579] tf possible? safely resize. Use Ocr IP needed/possible, 
[580] d. Handle graph label: 
[581] Long text; Ignore* or ocr and return as text 
[582] Short text (e,g. stock symbol/name); preserve label* but move It as 
needed* 

[583] Problem; range* label may not always be properly identified/handled 

for unknown format, 
[584] a. Resize graph; 

[585] Determine the maximal allowed size of the graph (after labels 

*waste*): With maximum scroll; with graph fully In screen; with graph and 
label fully in screen. 

[586] Determine which size to use, 

[587] Resize graph maintaining line continuity* and without thickening lines 

without necessity (consider graph as b/w), 
[588] Aspect ratio does not necessarily need to be maintained, 

^Bfo<± diagram; 
[589] HGdB shows a block diagram for the procedure 

b^— -Charts to Picture message 

^-toe^ae&ts - dhaagas toawfeisp: 
[590] a. Grid boundaries ignored (or just marks); b* Handling range: Ignore 
horizontal range; c. Range and labels; Use Ocr if possible (return: graph 
+ % MSFT* range 80-115*); d. Resizing graph is the same (with different 
parameters). 

4^-Convetston Algorithms s 

[591] Content for the system is b/w line art. The size and amount of text 
should allow applying moderate resizing and fit, or recognizing through 
OCR, Typical samples are available at *K:\QA\Image3anks\Comlcs\sllced 
eomics\b&w\ 

[592] a. Correct conversion of generic content within the one-block 
envelope, 

[593] b. Generic block Identification and splitting 
[594] c. Handling text; 
[595] I, Identifying text, 

[596] II, Identifying whether standard conversion would be ok, 
[597] ill. More moderate resizing + word reordering 
[598] Problems; 

[599] This may not always fit Do we split it to multiple images? 
[600] Identifying optimal resizing, 

[601] iv. Specialized resizing - to allow legibility after resizing, 
[60S] v. Recognizing with Ocr; 

[603] Evaluate and tone use of both scansoft and ParaScrlpt (cartoon text is 
closer to handprint) 
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[604) Appropriate preprocessing, 

[60S] Problems; ^ 

[606] Cartoon text is small and not neatly printed - Oct may fail, 

[607] Handling non-English text, 

[608] vl. Support getting text as input, 

[609] d, Recognfeing and cleaning non-relevant text 

[610] Rai9 shows a Block diagram without OCR, 
[6111 FIG,20 shows a Block diagram with OCR; 



[612] Ail limitations and stages for wbmp apply, 

[613] Further envelope constraints; 

[614] Text must be Ocr-able or given as input, 

[615] Text limited to 60 characters, 

[616] Image part content must fit in 28 pixel (height), 

tfe-An Example of a Combination of Techniques: Diflienng. and then Adap ting 
Photogra phs rwith different image material to WMBP, HMS, and/ or PM, format: 
a fr— "Generic" photo biaarizatioii 

[617] Typical scenes to be handled; ear(s), profile of person, full body, 

multiple faces/people, skyline, signs, trees, etc. 
[618] Desired output is the silhouette of the objects) in the image, an 

identifiable binarteation of the scene, 
[619] FIG,20 shows a Block diagram for this procedure 



[620] a. Emphasising boundaries, decreasing intra-surface changes 

(illumination, etc,), 
[621] b. Resizing 
[622] c. Identifying silhouettes, 
[623] d. Brightening background 

[624] e. Appropriate dithering within objects - when needed (only big 



[625] f. Combining the silhouettes with the surfaces, 
[626] A data set for tuning and evaluation should be collected from current 
image banks + web. 



[627] Problem; most scenes cant be converted te 28 pixels. It is usually 

difficult to identify important parts of the scene 
[628] Additional components; 
[629] "Central object* detection, 
[630] When that is not possible - additional resizing, 

gmr—VAS extended functionality 

F631] It is expected that a large portion of MMS traffic will be generated by 
VAS application such as; photo albums, game sites, news sites etc. The 
UCnGO MCS provides special functionality for such services; 

[632] 1, Watermark embedding/detection for digital rights management 
(DRH) - in order to track/block/limit the distribution of copyrighted 
content, a mechanism is provided to embed a watermark in any media 
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object. This watermark does not change the Image/sound of the media 
object in a perceptible way. It should be noted that this method does not 
interfere with other DRM mechanisms such as those provided already by 
Nokia (by special tags in WAP pages) and Ericsson (tags in content 
descriptor).. It provides an additional mechanism which is independent of 
external tags. 

[633] 2, Content Based Transcoding - the conversion of visual information 
(images, animations and video) to displays with limited color range, 
memory and resolution is enhanced by knowing the type of the visual 
content For example, the optimal conversion algorithm of a chart into a 
B/W display is different from the optimal algorithm for a natural 
photograph. See figure 4 for examples, 

[634] 3, Layered graphics input - the vast majority of professional graphic 
and photographic content is prepared using Adobe software tools such as 
PhotoShop* It is stored in a special format (PSD) which maintains the 
different layers of the image. Different layers can include the background, 
text, images added from other sources etc It is not unusual for a high 
end image for the publishing or web-publishing industries to include over 
SO separate layers. The MCS supports the PSD format as an input format, 
thereby preserving the original quality and enabling the content based 
detection module to handle each layer separately . 

tVdi— — Message interface 

[635] The MPS supports two distinct interfaces to the MMSC/external 
applications. 

[636] i, An XML-RPC/HTTP interface, enabling platform and operating 
system decoupling between the MMSC and the MPS. This interface, 
documented in the attached ICD, enables complete control over the 
manipulation and conversion operations of the media objects and works at 
the media level* 

[637] 2, A 3GPP standard, message-based Interface designed In to make the 
integration of the UCnGO MPS as standard as possible for the OEM MMSC 
integrator or the VAS provider. The Interface is based on the 
MM7/MM4/MM1 protocols. Using this interface, a complete unchanged 
MMS/Email message as received from the WAP GW/the other operator's 
MMSC/ the VAS can be sent as is to the MCS, and the response from the 
MCS can be sent as is to the recipient phone/MMSC/VAS server, 

[6383 Specifications; 

[639] The SMTP protocol (default port 25, configurable) or the HTTP POST 
protocol (configurable port) is used to transfer the message for 
processing. The message can be any standard MM1/MM4/MM7 message is 
defined In the 3GPPTS23.140 Release 5,20, document. The target 
device(s) type Identification can be performed in the following manners; 

[640] » The message header contains a set of (one or more) "X-MMS- 
UserAgent* or *X-MMS-UAprof or *X~MMS-Moder descriptors - in this 
case these descriptors are taken as representing the model types for the 
different intended recipients as they appear in the message in the TO; 
data field. 
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1641] The message header contains no extra information about the target 
devices* but an LDAP based user/device database has been configured to 
supply device parameters based on a user's MSISDN or email address. In 
this case the MCS performs an LDAP query far each target recipient 
specified in the "TO;* field of the message in order to find out the 
recipient terminal type. 



16431 The processed response in returned in one of the following manners; 

16441 The MCS can be configured to send the processed messages to a 
target SMTP server as MM7/MM4 messages. This way the MCS can sit 
between ,the external VAS/extemal MMSC and the local MMSC with no 
configuration changes, ■ 

F64S1 The ICS can be configured to send the processed messages via HTTP 
POST to a target server as an MMt message. This way the MCS can sit 
between the WW* GW and the MMSC MMS proxy, 

mm The processed response(s) will be sent in MIME multipart format, with 
the presentation layer and media objects converted based on the 
recipient device. For example, for a WAP phone the presentation layer will 
be in the text-wml MIME type, and images will be in GIF/WBMP format. 
The message will be submitted once per each target device, since the 
content for the different target devices is now different, having undergone 
conversion. So for example an incoming MM7 message targeted at four 
recipients wilt generate four MM7 messages with one recipient each. 

ffij, — .Presentation level conversion 

r6471 Effective multimedia presentation requires some information on the 
spatial and temporal relations between the different media objects 
presented. This functionality is performed by the presentation layer - 
HTML in web pages and Email, WML in WAP pages, SMIL in MMS 
messages. Some multimedia formate (e,g. EMS) and phones (e,g, Nokia 
3510, Nokia 7210) do not support an explicit presentation language, but 
rather display the different media objects according to their own pre- 
defined logic, 

\&m This means that In addition to the media objects conversion, the 
presentation level description has to be converted. In more complex cases 
fe g, when no presentation description language exists) the actual media 
conversion has to be changed to account for the presentation logic Some 
examples are: . _ 

r649] An image and accompanying text is to be sent to a WAP phone. By 
changing the image stee target one can guarantee that the text will be 
able to be viewed on the screen together wltii the image without scrolling. 
This requires knowledge of the phone's effective (versus physical) display 
stee, and control of the image stee in pixels, the WML description (e,g, the 
align^left* directive for the text), etc. Thus, the generated WML deck 
should contain the proper parameters. 

rsSQl An MMS message containing a cartoon with a set of images and 
associated text, alongside with SMIL description is sent to a Nokia 3510 
which does not support SMIL. There is no specific order for the media 
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objects - that is, the test Image (the cartoon's "punchline*) can be the 
first Image In the MMS message. Hence, if the SMIL description is just 
omitted, the received content will be worthless* The presentation level 
logic In this case has to order the media objects according to their desired 
display order as specified in the SMIL description* so that the Nokia user 
wilt get the cartoon in the proper sequence, 
[651] Specifications 

[652] The UCnGQ MCS provides for the presentation level conversion for the 
SMIL,MIME multipart, HTML.WML, EMS formats (see Rg. 37 for the 
supported conversion matrix). Of these, SMIL and MIME multipart are 
supported as input formats, and all are supported as output formats. 

[653] 

[654] The supported conversion operations include; 

[655] 1, Relative positioning of media objects based on target screen size. 

[656] 2. Formatting of text (Bold, Italic, underlined, size) *~ in EMS, 

WML,HTML, MIME/HTML 
[657] 3. Picture and animation smart compress - that is, the images and 

animations are adapted to fit in pixel size and total file size based on the 

dir&AJv&s of 

[658] 4, Audio smart compress for WAV/AMIVAAC, truncation for ringtones, 
[659] 5, Video frame rate reduction, resizing* 
[660] 6, Orientation change for images/video based on screen size. 
[661] 7. Different scaling based on parameters -use viewable area, use 

scrollable area, target application type (WAP/MMS etc). 
[662] Supported SMIL tags include: root-layout, region (and relevant tags), 

du*\ 

m&tr-Sigh Level Code for transcoding to a Picture Message 

[663] %This is an algorithm in pseudocode for converting any input image 

Into a B/W image with a maximum size of 72 by 2B pixels - 
[664] %The maximum size for a Nokia Smart Messaging Picture Message 
[665] function I4^newdither(fh,dorot), 

[666] % if dorot is specified the image is rotated by 90 degrees to make it fit 

the screen aspect ratio better* 
[667] I*imread(fn);I=*double(I)A56i 
[668] if existCdorotO^simrotatetl^O/nearest^jend; 

r©69i if 

lsgray(I),It=zeros(stze(I,l),size(I,2),3)Ut(;,i,l)=nn(:,:,2)«nit(:,:,3)=Ii 

I«tt;en<b %we make sure the image is 3 channel 
[670] %RGB image even if it started out as a grayscale image. 
[671] I0=rgb2hsv(l)^I0; t sv 

[672] fori«li3,I0(i,:,i)«fiiter2(ones(6,6)/36 t I0(.,i,i), % same , );end; 
[673] It(:,:,3)=*lt(.,:,3>0.75*10( ;,:,3);% We perform an unsharp operation 

in the luminance channel of the image 
[674] mx^max(max(It(.,J,3))); 
[675] mn*min(min(It(:,:,3))); 

[676] It(;,;,3)=histeq(It(i,:,3));%We stretch the histogram of the luminance 
channel 
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[677] l«hsv2rgb(lt);% And then convert back to RGB 

[678] I«tror«st2fe(l»73/(sfee(U))*tO^*s^(IA) steetl^ybiebufc'); 

[679] l=I(:,2:end,:)i WThe image Is resized, and eliminate the edge due to 

edge-scaling effects* 
[680] mx=max(It:)>; 
[681] mn*min(X(;))J 
[682] l«(I-mn)/(mx-mn); 
[683] tQ=rgb2hsv(I)#t«*0; 

[684] for i«l:3»I0(; t i,i)^filter2(ones(6,6)/36 r I0C;,: r i) t , same*)iendj 
[685] Htv,3)*4t(v,3)-G.$*10(:,l,3); 
[686] mx=max(max(lfc(:,:*3))); 
[687] mn=mintmin(It(:,:,3)))j 

[688] lt(uu3)==bisteqtK(;,i,3))i%{Itt: t : t 3)-mn)/(m^mn)i 

[689] I=hsv2rgbCIfc);%We repeat the same transformation for the resteee 

image with different unsharp parameters, 
[690] c^fio©rts&e(U)/2)J 
[691] U=I(cy-min(t3»cy-l);cy+min(14 t cy),; + :)J 
[69a] [t2,M]«rgb2ind(IlA'dither');% We reduce the image to 4 colors with 

dithering 

[693] 12«(rgb2gray(ind2rgb(I2,M)))j% Convert to grayscale 

[694] I2«(I2-min{ia(;)))/(max(l2(i»~min(l^(:)))J 

[69S] I3=^eros(size(ti»l)*7^3);t3t:,;4)«12jB(i,:^)«I2;l3<; t : t 3)«ia; 

[696] I4=dither(I3,[0*25 0.25 0,25? 0.7S 0,75 0.75$])>0; %finally we dither 

into 2 colors 

[697] return 

[698] 

[699] 

£&&£=IIigk Level Code for homogenized sharpening 

[700] 

[701] % smartjsharpenjhomo - Sharpening algorithm ) 

[702] % 

[703] % Usage: 

[704] % finaUmage-smart_sharpen J .homo(ini«aUmage»param) 

[70S] % 

[706] % Inputs: 

[707] % inttiaUmage - RGB 

[708] % param - parameters structure 

[709] % pararruRadius«tMU]j %LpHomo Window 

[710] % param.Power=[2,16]i %lpHomo Power 

[711] % param,WeightsKM3J %tpHomo Weight 

[712] % param*thr_gair»=20; %- threshold-tike gain for sharpening 

[713] % param.thrjp>ower=,3; 

[714] % param 4 thrjwindow=5j 

[715] % 

[716] % Outputs; 

[717] % finaUmage - RGB 

[718] % 
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[719] % 

[720] % See Also 
[721] % 

[722} % Written by Goldentayer Lev, March 2001 

[72:4] for eh-l:siMlnMaUmage,3) 
[725] mpie»tnltiaUm&get:*:^h)i 

[726] matVa!-param,Radlus(l);matrixDims « int32([matVal ,mat\tel ,matVaj 

,maWal ))J 
[727] 

Jl«double(iplmexCu^gstWomogentea^^ 

s*stngle(pan3rn,Poweitt)) ))i k 
[728] matVal«param,Radiust2);matrkDims « lnt32([matVal ,matVal ,matVai 

,mattfal ]); 

s,single{param,Power(2)) ))s 
[730] a0«param,Welghtstl)iai~param,Weights(2)i 
[731] mmpic*aO*Jl+al*J2)/(aO+at+l); 

[7323l=(l^min(ttO))/(ro®<l(^))-m^ClCi)))J „ ^ > t 

[733] K«ones(param,thr_window)/param,thr^^ 

[734] thr_mult=(max(w^tabs(mpi«> 

filter2(^mp5c))*paramvthrjgain4)*0))^param.thrjpowen 

[735] finaUmagef;,: ,f&)«mpic,*(l~thr„multH l,^rjrnultj 

[736] end 

[737] 

JV.K3. High level code for image color palette and size adaptation 
[739] 

[740] % This is a pseudocode implementation of a smart compression 



[741] % an algorithm that gets as its input an image file and outputs a 
version of the image in a specific format (GIF in this example) 

[742] % and with a maximum file stee that does not exceed the limitations 
dictated by the end device. 

[743] % This implementation iterates the number of colors in the 
quantization step until the file is small enough, 

[7441 function err»smartcompress(filename), 

[745] NumCoN[2 8 10 13 16 20 24 32]; WThese are the numbers of colors 
in the colormap - can range from 2-256 for a typical 256 color 

[746] %display. These quantised color number steps are dependent on the 
target device - one would choose diffemet steps for other devices 

[747] % in order to minimize the run time, 

[748] SteeThreshold«2720; % This is the upper threshold in bytes on the 

output file state for a T68 using a WAP browser, 
[749] Impath^^D^tmageMagickXtmageMagick-winSkV; 
[750] netpbmpath«*Di\netpbm\binV; 
[751] NumQuant^length(NumCol); 
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[752] l«imread(filename);t«doubte(I)/255; %we read the origin file and 

convert it to a floating point image* 
[753] Q=min(96/si^(I,2),100/stee(I,l)); 
[754] ll=zeros(stee(I))i 

[755] for Nli3^1(i»:,i)=«»tor2(onest3,3) t t(;,^i);sam^)iendi %We create a 
blurred (rectangular kernel) version of the original image, 

[756] I-tdoub!etl)-0.25/9*doubie(n));I«M,*a<0); %Andthen substractit 
from the original to create an unsharp operation. 

[757] lt»rgb2ycbcr(t)jIt=W^q(lt)gt=imadjust(lt t tO 1],[0.1 0*9]); 

%eonversion to lumlnanee/coior space to stretch the luminance histogram 

[7S8]lt(;,;4)«histegWi t i,i));It(:*:^)«imadjust(histeq(lM:i/>,2)),C0 

ltrcu o.8])j 

[759] W;*i,3)«imadjust(histeo,at(;ri»3)),[0 l],[0v2 0,B]); 

[760] I=(vebcr2rgb(It)); % we convert back to RGB colorspaee after 

stretching the luminance channel, ^ 
[761] I^histeq(I)jI«i:^0v75j %And then we further manipulate the 

[765] S ?iim^e(l ♦Qr , »teu^c , )i %We resiie to the target size in pixels (this 

could also be an iteration based on target file stee)* 
[763] imwrite(I^np^pgV*Quality\100)i 

[764] dosfflrnpath 'convert -compress 12W 1 pwd \lnp4P9 P^d *\inp,gif])j 
[765] des(fnetpbmpath 'giftopnm lnp.gif > inp*ppm']); %We convert the 

resized image to a ppm file so we can run the ppmquant algorithm 
[766] FileSi2e«2eros(l»NumQuant)i 
[767] MaxNum«2; 
[768] for t^liNumQuant-, 

[769] dos([netobmpath "ppmouant * num2str(NumCoi(i)) inp.ppm > 
outvPpm , 3); 

[770] dos([netpbmpath *ppmtogif outppm > outglf]); 

[772] RleSfce(i)-D,bytes} %We measure the file size of the output image* 
t773] If (FileStee(i)<Si2eThreshoid) t MaxNum«NumCol(i);endi % If it is 

valid (less than threshold) we update the maximum 
[774] %eolor number we could achieve 
[775] end; 

[776] disp(MaxNum); 

[777] dos([netpbmpath >pmquant * num2str(MaxNum) * lnp k ppm > 
out-ppm 1 "]); 

[778] dos([netpbmpath >pmtogif outppm > shidl,gif]); 

[779] dosCftp -sitransfenbat^ %we put the file in a directory with pre- 

source information (in this case a WML card deck) so that 
[780] %we can view the image on a target device 
[781] end; 



ff^4^mgh Level Code for «»fdp &£&xmtim ft> spe«M etdw hmthm 0-&>ty 

[784] 
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[785] function finaUmase^vSS^dith^rtJnltiaUmage) 
[786] % v332Ldfther — jphone dithering algorithm (for faees,ears etc) 
[78?) % This algorithm adapts to displays with a 3-3-2 color palotte (R,G t B). 
This is 

[788] % prevalent in Japans color handsets 
[7893 % Usage: 

[790] % finaUmage=v332_dither(inltteUmage) 
[791] % 

[792]% inputs: 

[793] % inittaljmage - RGB 

[794] % 

[795] % Outputs: 

[796] % ftaaUmage - 332 Image 

[797] % 

[798] % 

[799]% See Also 
[800] % 

[801] % Written by Goidentayer tew, March 2001 
[802] 

[803] for ch=l:3 
[804] 

[80S] initiaUmageugray ~ initiaijmaget: t : t ch) ; 
[806] 

[807] %mplc«tm2douWe(lrnresi2©(inlttaUrnagejgray,t92 98ybitubie))j 
[808] %mpte=im2double0mrestee(initiaUrnagejsray t t74 98]/bteubte*))j 
[809] resize_ratio=96/sfee(NtiaUmage»2)j 

mpie=im2doubte(imrestee(initlaUmag^^ 
[8t0] 

[811] mplc*unpair(mpic,tl 1 1 1])» 
[812] mpic«(mpic+histeg(rnpte))/2i 
[813] 

[814] matVal=2jmatrbd>ims « M32([matVal ,matVai t matVal ,matVat ])j 
[81S] 

J4=double0plmexCucngoLPHomogenlmtlon^,single(mpte) t rnatti^Dim 
s,singte(16) ))i 

[817] a=,3;mpte=(mpie+a*}4)/<i+a); 

[818] rgbjmage(:,:,ch)=rnpic; 

[819] end 

[820] 

[821] 

[822] |j*0} 
[823] for 11=1:8 
[824] fori2=l:8 
[825] for 13=1:4 
[826] jHj+l; 

[827] YMAP{&l)=(il~l)*(l/8)-Kl/l6)j 
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[829] YMAP(^3)-(l3^i)*((l-l/8)/3)+(l/16)i 

[830] end 

[831] end 

[832] end 

[8331 

[834] 

IV.K.5 High Level Code for JPEG artifacts removal 

[835] 

[836] function fin&Umage^jpegJftte!r(*nitiaOmage) 

[837] % jpeg JRtter - 3peg artifacts filter 

[838] % 

[839] % Usage: 

[840] % finaUinage«3pegJiiter(initiaUmage) 
[841] % 

[842] % Inputs; 

[843] % InitiaUmag^ - RGB 



[845] % Outputs; 

% ftaaUrnage ~ RGB 



[848] % 



[850] % See Ate© 
[851] % 

[852] % Written by Gotdentayer Lev, March 2001 
[853] 

[854] xpic«lm2doub»e(ta«SaUrnage); 

[855] for eh«l:stee(xpte»3) 

[856] xfilt(;,: t ch)«fttter2(ones(2)/4 t i?pict; r :,cn)); 

[857] end 

[858] for ch-l;stee(Kpic»3) 

[859] 3G-Kfilt(:,:,ch); , k 

[860] matVaN6imatrfodDim$ * int32([matVat ,matVal ,maWa* ,matVa! ])j 

f8611 

Jl*double(iplmexfumgoLPHomogenteationFP\single(xf1lt{:,:»ch)) t matrtxD 

[SsS^mltvSyimatrtxDIrns * int32([matVal ,roaWal ,matVal ,matVal ])} 

}2«doubleOPI^C^9 otpHomo 9^^^ 
txDims,stngle(16) ))i 
[864] a0=2;al=*,5;a2«*l; 

[865] xenh(;,;,ch)«(aO*30+al*31+a2*J2)/(aO+al+a2)t 
[866] end 

[867] for eh«l;stee(*pie,3) 

[868] finaUmage(:,;»ch)=w\ener2{xenh(; t udi) t [3 3])j 
[869] end 
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r®7oi 

[871] final imageKroaxtmta(finaUmage t l) t O) .*txpte+.2S)+xpfc*(l- 

xpte))/jU25; 
[872] 

IV.K.6. High Level Code for image content detection 



[875] function eontent£heH(varargin) 
[876] if(nargta=«0) 
[877] figure 

[878] set(gcf r mrne7Imagfc Content Deletion SheW); 
[879] settgcf;Men«Bar*/non^); 



[881] else 

[882] [injite, injpa^h} * «*ge*fiteC*.* 

[883] pause(*2)i 

[884] try 

[885] setOcf^Potnter'/watfihOf 

[886] contentJype=content„d^tect{in_fii^ Injptitn); 

[887] set(gc^PointerYanrow*); 



[889] catch 

[890] b%m 

[891] dtep(tesfc&"r)i 

[892] % dtepflnvalid m% mm&Yt 

[893] s^gtf,Wwter*, , »rrow , )j 

[894] end 

[89S] end 

[896] return 



[899] function €«»nt!entjcype«c©nten^deteetOrufi<et injpatft) 

[900] imagejrtame«DfU>&to*Mte 3» 

[901] irUrng«conteniLr©ad(imagejriame); 

[902] col„sat=content_color_sat(in„«mg); 

[903] bac^rounO^^^te^^^w^^OtU^S)? 

[904] dither_mea^re=con^t_dfther<lruirng); 

[90S] uniquej^tors^ntent.coteirsCirutog)? 



[907] ifCun^uejcotors=«2) „.-v~>-, . , .... 

[908] graOect«content jgra(JU^t(fitter2(on6s(5 t S)/2S t lrLjmg( v t l))) ; 

[909] end 

[910] lf(grad_vect(2)>gratL.ved<l)*l) 
[911] res=*PhotograpWcal Imaged 



[913] res^Synthetic Image? ^ 
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[914] |ftgradj^c*(l)>grad„vectC3)*i) 
[915] 

[916] if(dither_measure»*01) 

[9171 res=[res, Texture; ']; 

[918] end 

[919] end 

[920} fftc&Lsafc*©^) 

[921J res=[res, "Saturated Colors; *]; 

[9223 er, d 
[9233 end 



[925) 

[9263 iftuntquejsolors<*2) 
[927] res-[res/Blnary; *]; 

[929] W(ur»tquejcobrs<=2S6) 

[930] res«[res, num2str{unique_eolQrs) + ' Colors; ']; 

[9313 «>se 

[932] res*tres* *Wch Color; *J; 
[933] end 
[934] else 

[935] res^res, 'Grayscale; *]; 

[936] end 

[9373 

[938] iftbadl^round_veet(i)>Q,25) 

[939] res*[res, 'Background Color Deleted; ']; 

[940] end 



[945] 

[946] %grad ra vect 

[947] stat^rEdgei %num2str{gradLvect(l))/ \newune 
[948] 'Curved; *,num2$trtgracLvecfc(2)V \newttne 
[949] Smooth; >um2str{grad„vec*C3)y \newlln& V« 
[9S0] Texture; '^um2stir(dither„rneasurey \newline V» 
[951] 'Unique Colors! \num2str(unSque„eelorsy \newfine \.« 
[9521 'Color Saturation; Vnum2str(coUsstV \newllne V r 
[9531 'Background Area; \num2str(baekgroundjtfectti}) t * \newtine 
[954] 'Background Color; \num2str<backgroundjtfect(2:end))]; 

[955] 
[956] 

[957] %%ure(l), 
[958] *clf; 
[959] subpldt(l,2,l); 
[960] lmshow(injmg); 
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[961] t*tie((injfite]); 

[962] subplot(i,2,2); 

[963] settgcf/UnttsVnorrnattzed*) 

[964] imshow(ones(128 t 256)*0*9); 

[9653 tttle([\it\res]); 

[966] hh=te*t(20,70,$tat); 

[967] setfhVF&ntSteeVB); 



[9693 %dtspCkB*r) 

[9703 contentjype^res; 

[971] return 

[9713 

[973] 

[9743 function tnjmg^eontent read (frnagejftame) 

[9753 iftisemptytimageuname)), 

[976] imagejname^dolyparton.jpg^ 

[9773 end 

[978] 

[9793 aa«tmfinfoOmageL.name); 
[9823 eteft 

[9833 [A,Map]«tmread(imagejname); 
[984] inJmg«tm2d©ubte(ind2rgb(A,Map)); 
[9853 dear Aj dear Mapi 
[9863 end 

[9873 dfeptrtmage; *t!mage„nawe])f 
[988] return 



[991] function c©Lsat=content_cotoc.sQt{inJmg) 
[9923 dvjmg*std(injmg,l*3)j 
[9933 sst=prod( ske(lnjmg))i 

[994] %disp([ * color saturation: \num2$tr(sum(d\Orog{:))/£s*)3)* 
[99S] coLsat=surn(dvJmg(:))/ssU 
[9963 return 
[997] 

[998] function bad?gi^und^vect=content^background(irUrng) 

[999] nchannete*stee(injrng t 3); 

[1000] Img_area=sl2e(tn_img t l)*s^t^'f«9* 2 )i 

[1001] [A,M3=rgb2ind(inJmg,256); 

[1002] [aa*ind3=ma>t(lmWst(A»M))i 

[1003] background_veet(l)«aa/prod(stee(A))j 

[1004] bad<groundj*ect(2:l+ncbannete)«MClnd)j 

[1005] d1sp([ * Background steet \aaft>rod(steetA))3)i 

[1006] dtsp([ * Background color; *,num2str(M0nd))])j 

[1007] return 
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[1008] 
[10093 

[1010] function dittier_meQSum=contfint_dlther(lnJmg) 

[1011] nchannete=$t^(lnjmg,3); 

[1012] for(ch =lmdhanneis) 

[1013] tmpJms«trUm9(:»i,«h)i 

[1014] dither_scoire(ch)=mean2:t9bs(tmpJm9' 

medffit2(trop.Jmg,[3,3]))); 

[1015] end 

[1016] dither_m^sure»m^fitdlthec.sco)re); 

[1017] dl$p(rt>ttfcer seare\ nufn2sM^tfter_scofe)])i 

[1018] return 

[1020] function uniquej^torsacontentwCOlorsOnjmg) 

[1021] iftsumO«Jmg(0-0<Uw9tO>0)}) 

[1022] emunique(injmg); 

[1023] un^uej^lor$-length(M); 

[1024] else 

[1025] W(sum((inJmg(;)^mg(l))^2)} 

[1026] uniqttejt»lons«2; 

mm 



uniquejeotors-lj 
[1029] end 
[1030] end 
[1031] return 



[1033] 

[1034] function grad_vett-c^nten^grad_stat(Sn„jmg) 
[1035] nchanne»s=*stee0rUmg,3)j 
[1036] 
[1037] 



nhood^onestS t 5); 

[1039] pen_ups=ftoor(sum(nhoodti))*>7S)j 

tl040] perjdn«ce«(sum(hhoodt:))*^S); 

[1041] iftper_up<=per„dn) 

[1042] per„up^sum(nhhod(:))* 

[1043] perjdin=lj 

[1044] end 
[1045] 

[1046] thejmg-injmgj 
[1047] 

[1048] for(ch»l;ndhQnneis) t 

[1049] 9 rad(: ( :,ch)=(ordfiit2(the_twg(:»i t eh) t per_up»nhood)- 

ordfitt2(tiheJmg(t»:,ch) t per„dn*nhood)); 

[1050] end 
[1051] 

[1052] thr_edge=.085; 
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[1053] thr„nolse=*02; 

[1054] edgej9radHgrad> thr^edge); 

[1055] shape_grad-((grad<= thr„edge)&(grad>== thrjnoise)); 

[1056] nolse_grad=((9>r®d< thr_nolse)&(grad>~ thrjwatee)) J 

[1057] 

[1058] foKeh^lmehannels), 

[1059] msskjgrad(;,:^h)=(shQp©j9rad(t^»ch) & 

(fflter2(nhood,edge^rad(*,* f ch)}-^G)); 

[1060] end 
[1061] 

[1062] ssi~prod{ stee{lnjmg)); 

[1063] 

[1064] 

[1065] gradj*mct(i)«sum(edge^rad(:))/ssi; 

[1066] gradjmd(2)=sumCshap^rad(:))/ssU 

[1067] gr®d_v®ck(3)=sum(no?s^j9md(;))/ssU 

[1068] grad_vecfc(4)«sumCrnaskj9radti»/sstj 

[1069] return 

&^tf=~-Caching and Smart Forwarding 

[1070] Certain distribution circumstances in MMS lead to situation 
where the same content is sent to a large number of devices with similar 
or identical characteristics. Examples are: 

[1071] A VAS (e*g* a stock quote provider) sends an update to 
thousands of subscribers at the same time - tn this situation hundreds of 
them will have identical phones and therefore media conversion should 
not be repeated for each one* 

[1072] Superdlstribution - a user gets a funny message and forwards it 
to his buddies, who then send it onward to their friends etc* Since some of 
the recipients may use the same device, there is no need to keep 
converting the message again and again. Furthermore, the degradations 
caused by continuous conversion should and can be avoided* See for 
example figure 32- the 2nd recipient has a color enabled display, and 
hence should get a color image/video* This can oniy happen if smart 
forwarding is implemented, 

[1073] In short, caching means that when a new 

transcoding/conversion request arrives, the MPS looks in the cache to see 
if an identicaVprac«ea«y identical request for transcoding of the same 
media object into an identical/praetically identical device has been 
submitted in the past and if the result of this operation is still in the cache. 
If so, the USI of the object in the cache is returned as the transcoded 
result and the actual transcoding operation is avoided. 

[1074] 

[1075] Smart forwarding Is similar to regular caching, except that the 
MPS retrieves the cached transcoded result based on the original media 
object* That is, if a content request for object *B* (originally transcoded 
from object "A") to device *V is requested, the MPS retrieves the cached 
result of "A* transcoded into T, not of *B* transcoded into *T * 
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[3.076] Other modifications and variations to the invention will be 
apparent to those skilled in the art from the foregoing disclosure and 
teachings. Thus, while only certain embodiments of the invention have 
been specifically described herein, it will be apparent that numerous 
modifications may be made thereto without departing from the spirit and 
scope of the invention. 
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WHAT 3FS tfoAIMEftJS rf5 ^ , 

1. An MMS communication system for displaying Images on a display 
terminal of a mobile or portable communication device* the system 
comprising: 

m input adapted to receive pre-source information; 

a transmitter adapted to transmit the pre-source information; 

a server adapted to receive the transmitted pre-source information 

and further adapted to convert the pm-source information to source 

information suitable for display on the display terminal; and 

a source transmitter adapted to transmit the source information to the 

display terminal 

2. The system of claim 1, wherein the server further comprises a 
display characteristics Identifier adapted to Identify display 
characteristics of the display terminal. 

3. The system of daim 1, wherein the server further comprises a pre- 
source transcoder adapted to transcode pre-souree information to 
source information. 

4. The system of daim 1, wherein the served comprises at least one 
source transcoder adapted to transcode a first source information in a 
first format to a second source information in a second format, 

5* The system of claim 4, wherein the second format corresponds to a 
display format for the display terminal, 

6, A method for enabling communication of MMS information suitable 
for display on a mobile or portable communication terminal, 
comprising: 

inputting pre-source information; 
transcoding pra-souree information into source information; 
transcoding source information another source format suitable for 
display on the target display terminal; Jt , 

transmitting the transcoded source information to the target display 
terminal; t . 

displaying the transmitted and transcoded source information on the 
display space of a target display terminal in a format most sulteble for 
that particular display terminal. 
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Figure XX: WatermarMng algorithm for B/W images (picture messages, WBMP) 
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Figure XX: Layer reduction and its effect on the message appearance 
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Figure XX: content based conversion for charts (color line-art) 
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Figure XX: content based conversion for cartoons (B&W Line-art) 
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Figure: cases where caching and smart-forwarding are required. 
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Specifications 



Implementation: The caching and smart forwarding system is based on a web-caching 
mechanism - converted and original media objects are stored in a specially configured 
set of web-caches, and are retrieved by URI. The basic operation scenario is depicted 
in the figures below. 
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Figure; Sequence of operations for a cached transaction 

Algorithmic Cache Sequence 



. For incoming object A {after SMF) + filtered Target Parameters 
calculate MD5 "C" plus target MIME type "D". 

• Perform HTTP HEAD request to check if in cache, request filename is 
"CVD". 

• If not in cache, perform transcode, perform HTTP PUT for transcode 
result in cache. 

• Return URI. 

Figure: sequence of operations for caching 
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fry 31 
Algorithmic SMF Sequence 

• For incoming object A calculate MD5, extract internal hash K C", MIME 
type "D" from SQL database (indexed query). 

• If new object, put in database as original content, perform HTTP PUT 
for original object in cache. 

• If existing object, perform HTTP HEAD request to check if in cache. 
Object filename is "C"."D ,! . Update last requested counter. 

• If in cache, return UR1. 

• If not in cache, put the new input object A as file with the original 
content hash value "C" as the file name. 

Figure: sequence of operations for smart-forwarding 

Hash Database 

• Any SQL database can be used, e.g. MySQL 

• Single indexed table {index on key value)/ double indexed table {last 
requested update). 

• Periodic cleanup required of least accessed objects. 

• Table can reside in RAM - e.g. for 10 Million cached messages, less 
than 4Gb of RAM required. 

• Can be implemented as an indexed linked list with no database if 
database performance issues arise. 

• Database scaling is simple based on hash value partitioning {e.g. 
server number 1 handles hash values less than TBD etc.). 

Figure: Hash database connecting transcoded objects and original objects 
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Figure XX: MM1 message interface to the UCnGO MCS 
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Figure XX: MM7 message interface to the UCnGO MCS 
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S: <wait for connection on TCP port 25> 

C: <open connection to server> 

S; 220rnmscj*cngomcs,net Simple Mai Transfer Service Ready 

C: EHLO cnn.com 

S: 250 mm$e,ucngomcs,net 

S: 250 SIZE 150000 

C: MAIL FROM: <(WW8HP> 

S: 250 OK 

C: RCPT TO: < *35084012^567®mmst. operator.net > 

S: 250 OK 

C: DATA 

S: 354 Startmaii input ; end with <CR><LP>.< CR><LF> 

C: Date: 01 November 2001 00:00 EST 

C: from: ■ com 

C: to: + 35084D1234567 tonsc.ooerator.net 

C: subject "Newscast with text and animated gjjj ■ 

C: Content-type: multipart/mke d 

C: X^terMessage-Ciass: personal 

C: X-Mgj^Expoy 05 November 2002 10:00 EST 

C: X-Ms^r^eiivery-time: 05 November 2002 09:00 EST 

C: X-lto-Pnority: Normal 

C: ^^JPartJJ840522,99773479?3d3 

C: Content-Type: text/plain; jteOSJgB&w^ 

C: Unisys launches MMS video services in Portugal 

C: -^_=JParLD_?84052i997734797363 

C: Content-Type: m^gt/g^mm'JM^^ 1 

C: Content-Disposition attachment; WmBSmM^SX^iM 

C: «JarUL?840522.997734797M3 

C: Content-Type: image/GIF 

C: Content-Transfer-Enco tiiqg: b ased4 

C: jlCY MTAyMQ CNMkagDE0Ny4KMTcuMjA5LjkSL 1 REUS U 9SVB2NACPgJ E3Lj 



C: bmcgdGHLICJQcmlvcml0eSlgZmLLbGQuIEL01GL2lHBldCBhcyAiTG9SIi4NC 

C: <CK><LF>.< CRxLF> 
S: 250 OK 
C: QUIT 

S: 221 mmsc.ucngomcs.net Service closing transmission channel 
Figure: Sample MMS submission for transcoding from a VAS through MM7 
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